Recently there has been a significant surge in multimodal learning in terms
of both image-to-text and text-to-image generation. However, the success is
typically limited to English, leaving other languages largely behind. Building
a competitive counterpart in other languages is highly challenging due to the
low-resource nature of non-English multimodal data (i.e., lack of large-scale,
high-quality image-text data). In this work, we propose MPM, an effective
training paradigm for training large multimodal models in low-resource
languages. MPM demonstrates that Multilingual language models can Pivot
zero-shot Multimodal learning across languages. Specifically, based on a strong
multilingual large language model, multimodal models pretrained on English-only
image-text data can well generalize to other languages in a zero-shot manner
for both image-to-text and text-to-image generation, even surpassing models
trained on image-text data in native languages. Taking Chinese as a practice of
MPM, we build large multimodal models VisCPM in image-to-text and text-to-image
generation, which achieve state-of-the-art (open-source) performance in
Chinese. To facilitate future research, we open-source codes and model weights
at this https URL

提出 MPM，一种在低资源语言中训练大型多模态模型的有效训练范例，通过多语言语言模型对仅在英语图像文本数据上预训练的多模态模型实现跨语言的零样本学习，取得了比母语语言的图像文本数据训练的模型还要好的图像文本生成性能。