Human intelligence is multimodal; we integrate visual, linguistic, and
acoustic signals to maintain a holistic worldview. Most current pretraining
methods, however, are limited to one or two modalities. We present i-Code, a
self-supervised pretraining framework where users may flexibly combine the
modalities of vision, speech, and language into unified and general-purpose
vector representations. In this framework, data from each modality are first
given to pretrained single-modality encoders. The encoder outputs are then
integrated with a multimodal fusion network, which uses novel attention
mechanisms and other architectural innovations to effectively combine
information from the different modalities. The entire system is pretrained
end-to-end with new objectives including masked modality unit modeling and
cross-modality contrastive learning. Unlike previous research using only video
for pretraining, the i-Code framework can dynamically process single, dual, and
triple-modality data during training and inference, flexibly projecting
different combinations of modalities into a single representation space.
Experimental results demonstrate how i-Code can outperform state-of-the-art
techniques on five video understanding tasks and the GLUE NLP benchmark,
improving by as much as 11% and demonstrating the power of integrative
multimodal pretraining.

该论文提出了一种自我监督的预训练框架 ——i-Code，用户可以将视觉、语音和语言的模态灵活地组合成统一的多用途向量表示，通过融合网络使用新颖的注意机制和其他架构创新来从不同模态中有效地组合信息，实验证明 i-Code 可以优于现有的技术在五个视频理解任务和 GLUE NLP benchmarks 上取得最先进的表现。