We present a new large-scale multilingual video description dataset, VATEX, which contains over 41,250 videos and 825,000 captions in both English and Chinese. Among the captions, there are over 206,000 English-Chinese parallel translation pairs. Compared to the widely-used MSR-VTT dataset, VATEX is multilingual, larger, linguistically complex, and more diverse in terms of both video and natural language descriptions. We also introduce two tasks for video-and-language research based on VATEX: (1) Multilingual Video Captioning, aimed at describing a video in various languages with a compact unified captioning model, and (2) Video-guided Machine Translation, to translate a source language description into the target language using the video information as additional spatiotemporal context. Extensive experiments on the VATEX dataset show that, first, the unified multilingual model can not only produce both English and Chinese descriptions for a video more efficiently, but also offer improved performance over the monolingual models. Furthermore, we demonstrate that the spatiotemporal video context can be effectively utilized to align source and target languages and thus assist machine translation. In the end, we discuss the potentials of using VATEX for other video-and-language research.

我们提出了一个新的大规模多语言视频描述数据集 VATEX, 其中包含超过41,250个视频和825,000条英文和中文字幕，拥有超过206,000个英中平行翻译对。我们还基于 VATEX 引入了两项视频与语言研究任务：（1）多语言视频字幕生成，旨在使用紧凑的统一字幕模型以各种语言描述视频，（2）视频引导机器翻译，使用视频信息作为附加时空上下文将源语言描述翻译成目标语言。VATEX数据集的广泛实验表明，该统一多语言模型不仅可以更高效地生成视频的英文和中文描述，而且可以提供比单语言模型更好的性能。此外，我们还证明，时空视频上下文可以有效地用于对齐源语言和目标语言，从而帮助机器翻译。最后，我们讨论了使用VATEX进行其他视频与语言研究的潜力。

VATEX：一个大规模、高质量、多语种的视频与语言研究数据集