We present Chameleon, a family of early-fusion token-based mixed-modal models
capable of understanding and generating images and text in any arbitrary
sequence. We outline a stable training approach from inception, an alignment
recipe, and an architectural parameterization tailored for the early-fusion,
token-based, mixed-modal setting. The models are evaluated on a comprehensive
range of tasks, including visual question answering, image captioning, text
generation, image generation, and long-form mixed modal generation. Chameleon
demonstrates broad and general capabilities, including state-of-the-art
performance in image captioning tasks, outperforms Llama-2 in text-only tasks
while being competitive with models such as Mixtral 8x7B and Gemini-Pro, and
performs non-trivial image generation, all in a single model. It also matches
or exceeds the performance of much larger models, including Gemini Pro and
GPT-4V, according to human judgments on a new long-form mixed-modal generation
evaluation, where either the prompt or outputs contain mixed sequences of both
images and text. Chameleon marks a significant step forward in a unified
modeling of full multimodal documents.

Chameleon 是一种早期融合的基于令牌的混合模态模型系列，能够理解和生成任意顺序的图像和文本。它在图像问答、图像描述、文本生成、图像生成和长格式混合模态生成等任务上展现了广泛和通用的能力，包括在图像描述任务中达到了最先进的性能水平，同时在纯文本任务中超越了 Llama-2，与 Mixtral 8x7B 和 Gemini-Pro 等模型竞争，并且能够生成非平凡的图像，所有这些能力都集成在一个模型中。它还在全新的长格式混合模态生成评估中，通过人类评判匹配或超越了更大模型，包括 Gemini Pro 和 GPT-4V，该评估中的提示或输出包含混合的图像和文本序列。Chameleon 在统一建模完整多模态文档方面迈出了重要的一步。