This paper studies zero-shot cross-lingual transfer of vision-language models. Specifically, we focus on multilingual text-to-video search and propose a Transformer-based model that learns contextualized multilingual multimodal embeddings. Under a zero-shot setting, we empirically demonstrate that performance degrades significantly when we query the multilingual text-video model with non-English sentences. To address this problem, we introduce a multilingual multimodal pre-training strategy, and collect a new multilingual instructional video dataset (MultiHowTo100M) for pre-training. Experiments on VTT show that our method significantly improves video search in non-English languages without additional annotations. Furthermore, when multilingual annotations are available, our method outperforms recent baselines by a large margin in multilingual text-to-video search on VTT and VATEX; as well as in multilingual text-to-image search on Multi30K. Our model and Multi-HowTo100M is available at http://github.com/berniebear/Mutli-HT100M.

本文研究了视觉语言模型的零样本跨语言迁移。我们专注于多语言文本到视频搜索，并提出了一种基于Transformer模型的方法来学习上下文相关的多语言多模态嵌入。在零样本设置下，我们在用非英语句子查询多语言文本-视频模型时发现性能显著下降。为了解决这个问题，我们引入了多语言多模态预训练策略，并收集了一个新的多语言教学视频数据集（MultiHowTo100M）进行预训练。在VTT上的实验表明，我们的方法显著提高了非英语语言的视频搜索效果，而无需额外的注释。此外，当有多语言注释时，我们的方法在VTT和VATEX上的多语言文本到视频搜索以及Multi30K上的多语言文本到图像搜索方面大幅优于最近的基准线。

多语言多模态预训练用于零样本跨语言视觉语言模型迁移