Most existing cross-modal language-to-video retrieval (VR) research focuses
on single-modal input from video, i.e., visual representation, while the text
is omnipresent in human environments and frequently critical to understand
video. To study how to retrieve video with both modal inputs, i.e., visual and
text semantic representations, we first introduce a large-scale and cross-modal
Video Retrieval dataset with text reading comprehension, TextVR, which contains
42.2k sentence queries for 10.5k videos of 8 scenario domains, i.e., Street
View (indoor), Street View (outdoor), Games, Sports, Driving, Activity, TV
Show, and Cooking. The proposed TextVR requires one unified cross-modal model
to recognize and comprehend texts, relate them to the visual context, and
decide what text semantic information is vital for the video retrieval task.
Besides, we present a detailed analysis of TextVR compared to the existing
datasets and design a novel multimodal video retrieval baseline for the
text-based video retrieval task. The dataset analysis and extensive experiments
show that our TextVR benchmark provides many new technical challenges and
insights from previous datasets for the video-and-language community. The
project website and GitHub repo can be found at
this https URL and
this https URL, respectively.

提出了一种新的跨模态视频检索数据集 TextVR，它包含了八个场景领域的 10.5k 个视频和 42.2k 个查询语句，并介绍了一种统一的跨模态模型，有效地将文本和视觉语义信息融合来实现视频检索任务。该数据集和我们提出的跨模态检索方法为视频和语言研究领域提供了许多新的技术挑战和洞见。