We introduce CM3, a family of causally masked generative models trained over
a large corpus of structured multi-modal documents that can contain both text
and image tokens. Our new causally masked approach generates tokens left to
right while also masking out a small number of long token spans that are
generated at the end of the string, instead of their original positions. The
casual masking object provides a type of hybrid of the more common causal and
masked language models, by enabling full generative modeling while also
providing bidirectional context when generating the masked spans. We train
causally masked language-image models on large-scale web and Wikipedia
articles, where each document contains all of the text, hypertext markup,
hyperlinks, and image tokens (from a VQVAE-GAN), provided in the order they
appear in the original HTML source (before masking). The resulting CM3 models
can generate rich structured, multi-modal outputs while conditioning on
arbitrary masked document contexts, and thereby implicitly learn a wide range
of text, image, and cross modal tasks. They can be prompted to recover, in a
zero-shot fashion, the functionality of models such as DALL-E, GENRE, and HTLM.
We set the new state-of-the-art in zero-shot summarization, entity linking, and
entity disambiguation while maintaining competitive performance in the
fine-tuning setting. We can generate images unconditionally, conditioned on
text (like DALL-E) and do captioning all in a zero-shot setting with a single
model.

介绍了 CM3 模型，是一种因果掩模生成模型，用于训练多模态文件的大型语料库，可生成结构化文本或图像，支持双向上下文依赖，具有接近最先进性能的能力，能够通过单个模型完成零 - shot 任务的多模态生成、文本摘要、实体链接、实体消岐等。