Recent years have witnessed a big convergence of language, vision, and
multi-modal pretraining. In this work, we present mPLUG-2, a new unified
paradigm with modularized design for multi-modal pretraining, which can benefit
from modality collaboration while addressing the problem of modality
entanglement. In contrast to predominant paradigms of solely relying on
sequence-to-sequence generation or encoder-based instance discrimination,
mPLUG-2 introduces a multi-module composition network by sharing common
universal modules for modality collaboration and disentangling different
modality modules to deal with modality entanglement. It is flexible to select
different modules for different understanding and generation tasks across all
modalities including text, image, and video. Empirical study shows that mPLUG-2
achieves state-of-the-art or competitive results on a broad range of over 30
downstream tasks, spanning multi-modal tasks of image-text and video-text
understanding and generation, and uni-modal tasks of text-only, image-only, and
video-only understanding. Notably, mPLUG-2 shows new state-of-the-art results
of 48.0 top-1 accuracy and 80.3 CIDEr on the challenging MSRVTT video QA and
video caption tasks with a far smaller model size and data scale. It also
demonstrates strong zero-shot transferability on vision-language and
video-language tasks. Code and models will be released in
this https URL.

该论文提出了一种新的多模态预训练统一范式 mPLUG-2，该范式具有模块化设计和通用模块的共享，可以处理多种任务，包括文本，图像和视频的多模态理解和生成，达到了最先进的结果。

mPLUG-2：跨文本、图像和视频的模块化多模态基础模型

mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video

Although deep networks have recently emerged as the model of choice for many
computer vision problems, in order to yield good results they often require
time-consuming architecture search. To combat the complexity of design choices,
prior work has adopted the principle of modularized design which consists in
defining the network in terms of a composition of topologically identical or
similar building blocks (a.k.a. modules). This reduces architecture search to
the problem of determining the number of modules to compose and how to connect
such modules. Again, for reasons of design complexity and training cost,
previous approaches have relied on simple rules of connectivity, e.g.,
connecting each module to only the immediately preceding module or perhaps to
all of the previous ones. Such simple connectivity rules are unlikely to yield
the optimal architecture for the given problem.
In this work we remove these predefined choices and propose an algorithm to
learn the connections between modules in the network. Instead of being chosen a
priori by the human designer, the connectivity is learned simultaneously with
the weights of the network by optimizing the loss function of the end task
using a modified version of gradient descent. We demonstrate our connectivity
learning method on the problem of multi-class image classification using two
popular architectures: ResNet and ResNeXt. Experiments on four different
datasets show that connectivity learning using our approach yields consistently
higher accuracy compared to relying on traditional predefined rules of
connectivity. Furthermore, in certain settings it leads to significant savings
in number of parameters.

本文提出了一种利用模块化设计原则、通过学习连接来自动优化神经网络架构的算法，在多类图像分类任务中，使用 ResNet 和 ResNeXt 两种常见架构进行实验，结果表明与传统模块连接规则相比，学习连接的方法可以显著提高模型准确性并实现参数数量的节省。