People perceive the world with multiple senses (e.g., through hearing sounds,
reading words and seeing objects). However, most existing AI systems only
process an individual modality. This paper presents an approach that excels at
handling multiple modalities of information with a single model. In our
"{SkillNet}" model, different parts of the parameters are specialized for
processing different modalities. Unlike traditional dense models that always
activate all the model parameters, our model sparsely activates parts of the
parameters whose skills are relevant to the task. Such model design enables
SkillNet to learn skills in a more interpretable way. We develop our model for
five modalities including text, image, sound, video and code. Results show
that, SkillNet performs comparably to five modality-specific fine-tuned models.
Moreover, our model supports self-supervised pretraining with the same sparsely
activated way, resulting in better initialized parameters for different
modalities. We find that pretraining significantly improves the performance of
SkillNet on five modalities, on par with or even better than baselines with
modality-specific pretraining. On the task of Chinese text-to-image retrieval,
our final system achieves higher accuracy than existing leading systems
including Wukong{ViT-B} and Wenlan 2.0 while using less number of activated
parameters.

本文提出了一个能够优秀地处理多模态信息的模型，名为 SkillNet，其不同的参数部分专门用于处理不同的信息模态，并且可以稀疏激活模型参数的相关技能。我们使用自监督预训练方式对五种模态进行了开发，结果表明该模型表现与五个模态特定的微调模型相当。在汉语文本 - 图片检索任务中，我们的系统使用较少的激活参数数目，就实现了比现有优秀的 Wukong {ViT-B} 和 Wenlan 2.0 更高的准确性。