Large-scale pretrained foundation models have been an emerging paradigm for
building artificial intelligence (AI) systems, which can be quickly adapted to
a wide range of downstream tasks. This paper presents mPLUG, a new
vision-language foundation model for both cross-modal understanding and
generation. Most existing pre-trained models suffer from the problems of low
computational efficiency and information asymmetry brought by the long visual
sequence in cross-modal alignment. To address these problems, mPLUG introduces
an effective and efficient vision-language architecture with novel cross-modal
skip-connections, which creates inter-layer shortcuts that skip a certain
number of layers for time-consuming full self-attention on the vision side.
mPLUG is pre-trained end-to-end on large-scale image-text pairs with both
discriminative and generative objectives. It achieves state-of-the-art results
on a wide range of vision-language downstream tasks, such as image captioning,
image-text retrieval, visual grounding and visual question answering. mPLUG
also demonstrates strong zero-shot transferability when directly transferred to
multiple video-language tasks.

mPLUG 是一个新的视觉语言基础模型，通过新颖的跨模态 skip-connections 有效缓解了现有预训练模型由于跨模态对齐中长的视觉序列带来的信息不对称和低计算效率等问题，通过对大规模图像 - 文本对的端到端预训练，具有良好的视觉和语言能力，并在图像字幕生成、图像 - 文本检索、视觉定位和视觉问答等一系列视觉语言场景下实现了最先进的结果，并且具有很强的零样本学习能力。