Video-Language Pre-training models have recently significantly improved
various multi-modal downstream tasks. Previous dominant works mainly adopt
contrastive learning to achieve global feature alignment across modalities.
However, the local associations between videos and texts are not modeled,
restricting the pre-training models' generality, especially for tasks requiring
the temporal video boundary for certain query texts. This work introduces a
novel text-video localization pre-text task to enable fine-grained temporal and
semantic alignment such that the trained model can accurately perceive temporal
boundaries in videos given the text description. Specifically, text-video
localization consists of moment retrieval, which predicts start and end
boundaries in videos given the text description, and text localization which
matches the subset of texts with the video features. To produce temporal
boundaries, frame features in several videos are manually merged into a long
video sequence that interacts with a text sequence. With the localization task,
our method connects the fine-grained frame representations with the word
representations and implicitly distinguishes representations of different
instances in the single modality. Notably, comprehensive experimental results
show that our method significantly improves the state-of-the-art performance on
various benchmarks, covering text-to-video retrieval, video question answering,
video captioning, temporal action localization and temporal moment retrieval.
The code will be released soon.

本研究提出了一种文本 - 视频本地化预文本任务，以实现细粒度的时间和语义对齐，从而让训练模型能够准确感知给定文本描述的视频时间边界，并且实验结果表明该方法在各种基准测试中显著提高了最先进的性能。