We present CM3Leon (pronounced "Chameleon"), a retrieval-augmented,
token-based, decoder-only multi-modal language model capable of generating and
infilling both text and images. CM3Leon uses the CM3 multi-modal architecture
but additionally shows the extreme benefits of scaling up and tuning on more
diverse instruction-style data. It is the first multi-modal model trained with
a recipe adapted from text-only language models, including a large-scale
retrieval-augmented pre-training stage and a second multi-task supervised
fine-tuning (SFT) stage. It is also a general-purpose model that can do both
text-to-image and image-to-text generation, allowing us to introduce
self-contained contrastive decoding methods that produce high-quality outputs.
Extensive experiments demonstrate that this recipe is highly effective for
multi-modal models. CM3Leon achieves state-of-the-art performance in
text-to-image generation with 5x less training compute than comparable methods
(zero-shot MS-COCO FID of 4.88). After SFT, CM3Leon can also demonstrate
unprecedented levels of controllability in tasks ranging from language-guided
image editing to image-controlled generation and segmentation.

CM3Leon 是一种检索增强的令牌化、仅解码器的多模式语言模型，能够生成和填充文本和图像，通过多模式体系结构实现规模化和调优多样化指导式数据的极大益处，附加了大规模检索增强预训练阶段和第二个多任务监督微调阶段。它是一种通用模型，可以进行文本到图像和图像到文本的生成，可生成高质量输出的自包含对比解码方法。广泛实验证明了这种方法对多模态模型的高效性，CM3Leon 在文本到图像生成方面实现了最先进的性能，训练计算量比可比方法少 5 倍（零样本 MS-COCO FID 为 4.88）。经过微调后，CM3Leon 还可以展示在从语言引导的图像编辑到图像控制生成和分割的任务中前所未有的可控水平。