We propose an efficient method to ground pretrained text-only language models
to the visual domain, enabling them to process and generate arbitrarily
interleaved image-and-text data. Our method leverages the abilities of language
models learnt from large scale text-only pretraining, such as in-context
learning and free-form text generation. We keep the language model frozen, and
finetune input and output linear layers to enable cross-modality interactions.
This allows our model to process arbitrarily interleaved image-and-text inputs,
and generate free-form text interleaved with retrieved images. We achieve
strong zero-shot performance on grounded tasks such as contextual image
retrieval and multimodal dialogue, and showcase compelling interactive
abilities. Our approach works with any off-the-shelf language model and paves
the way towards an effective, general solution for leveraging pretrained
language models in visually grounded settings.

该研究提出了一种有效的方法，将预训练的纯文本语言模型转移到视觉领域，使其能够处理和生成任意交错的图像和文本数据，并在上下文图像检索和多模态对话等方面实现了强有力的效果。