When trained at sufficient scale, auto-regressive language models exhibit the
notable ability to learn a new language task after being prompted with just a
few examples. Here, we present a simple, yet effective, approach for
transferring this few-shot learning ability to a multimodal setting (vision and
language). Using aligned image and caption data, we train a vision encoder to
represent each image as a sequence of continuous embeddings, such that a
pre-trained, frozen language model prompted with this prefix generates the
appropriate caption. The resulting system is a multimodal few-shot learner,
with the surprising ability to learn a variety of new tasks when conditioned on
examples, represented as a sequence of multiple interleaved image and text
embeddings. We demonstrate that it can rapidly learn words for new objects and
novel visual categories, do visual question-answering with only a handful of
examples, and make use of outside knowledge, by measuring a single model on a
variety of established and new benchmarks.

通过对齐图像和标题数据，我们训练一个视觉编码器将每个图像表示为一系列连续的嵌入，并使用预先训练的冻结语言模型来生成相应的标题，从而将这种少量样本学习能力转移至多模态设置。该系统是一种多模态少样本学习模型，具有学习各种新任务的惊人能力，如用只有少数几个样例进行视觉问答，或者利用外部知识。