Multi-modal pretraining for learning high-level multi-modal representation is
a further step towards deep learning and artificial intelligence. In this work,
we propose a novel model, namely InterBERT (BERT for Interaction), which is the
first model of our series of multimodal pretraining methods M6
(MultiModality-to-MultiModality Multitask Mega-transformer). The model owns
strong capability of modeling interaction between the information flows of
different modalities. The single-stream interaction module is capable of
effectively processing information of multiple modalilties, and the two-stream
module on top preserves the independence of each modality to avoid performance
downgrade in single-modal tasks. We pretrain the model with three pretraining
tasks, including masked segment modeling (MSM), masked region modeling (MRM)
and image-text matching (ITM); and finetune the model on a series of
vision-and-language downstream tasks. Experimental results demonstrate that
InterBERT outperforms a series of strong baselines, including the most recent
multi-modal pretraining methods, and the analysis shows that MSM and MRM are
effective for pretraining and our method can achieve performances comparable to
BERT in single-modal tasks. Besides, we propose a large-scale dataset for
multi-modal pretraining in Chinese, and we develop the Chinese InterBERT which
is the first Chinese multi-modal pretrained model. We pretrain the Chinese
InterBERT on our proposed dataset of 3.1M image-text pairs from the mobile
Taobao, the largest Chinese e-commerce platform. We finetune the model for
text-based image retrieval, and recently we deployed the model online for
topic-based recommendation.

该论文提出了一种基于多模态预训练的新型模型 InterBERT，通过预训练的方式实现了多个任务，包括掩码片段建模、掩码区域建模和图像与文本匹配，并在视觉和语言下游任务上进行了 fine-tuning，最终实现了基于主题的推荐和基于文本的图像检索。