Vision language pre-training aims to learn alignments between vision and
language from a large amount of data. We proposed multi-grained vision language
pre-training, a unified approach which can learn vision language alignments in
multiple granularity. This paper advances the proposed method by unifying image
and video encoding in one model and scaling up the model with large-scale data.
We present X$^2$-VLM, a pre-trained VLM with a modular architecture for both
image-text tasks and video-text tasks. Experiment results show that X$^2$-VLM
performs the best on base and large scale for both image-text and video-text
tasks, making a good trade-off between performance and model scale. Moreover,
we show that the modular design of X$^2$-VLM results in high transferability
for X$^2$-VLM to be utilized in any language or domain. For example, by simply
replacing the text encoder with XLM-R, X$^2$-VLM outperforms state-of-the-art
multilingual multi-modal pre-trained models without any multilingual
pre-training. The code and pre-trained models will be available at
this http URL.

这篇论文提出了一种名为 multi-grained vision language pre-training 的视觉语言联合预训练方法，它可以在多个粒度上学习视觉语言对齐。该论文还提出了一个名为 X$^2$-VLM 的预训练模型，它采用了模块化架构，可在图像文本任务和视频文本任务中实现最佳性能和模型规模间的良好平衡，并显示了其高可转移性，可以在任何语言或领域中使用。

X$^2$-VLM：图像和语言任务的多功能预训练模型

X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks

This paper surveys vision-language pre-training (VLP) methods for multimodal
intelligence that have been developed in the last few years. We group these
approaches into three categories: ($i$) VLP for image-text tasks, such as image
captioning, image-text retrieval, visual question answering, and visual
grounding; ($ii$) VLP for core computer vision tasks, such as (open-set) image
classification, object detection, and segmentation; and ($iii$) VLP for
video-text tasks, such as video captioning, video-text retrieval, and video
question answering. For each category, we present a comprehensive review of
state-of-the-art methods, and discuss the progress that has been made and
challenges still being faced, using specific systems and models as case
studies. In addition, for each category, we discuss advanced topics being
actively explored in the research community, such as big foundation models,
unified modeling, in-context few-shot learning, knowledge, robustness, and
computer vision in the wild, to name a few.

本文研究多模态智能领域的视觉 - 语言预训练方法，分为三类分类，包含图像 - 文本、核心计算机视觉和视频 - 文本任务，针对每类任务，提出了针对性的方法，分别探究了研究进展和存在的挑战并讨论了更先进的主题。