We present Chameleon, a family of early-fusion token-based mixed-modal models
capable of understanding and generating images and text in any arbitrary
sequence. We outline a stable training approach from inception, an alignment
recipe, and an architectural parameterization tailored for the early-fusion,
token-based, mixed-modal setting. The models are evaluated on a comprehensive
range of tasks, including visual question answering, image captioning, text
generation, image generation, and long-form mixed modal generation. Chameleon
demonstrates broad and general capabilities, including state-of-the-art
performance in image captioning tasks, outperforms Llama-2 in text-only tasks
while being competitive with models such as Mixtral 8x7B and Gemini-Pro, and
performs non-trivial image generation, all in a single model. It also matches
or exceeds the performance of much larger models, including Gemini Pro and
GPT-4V, according to human judgments on a new long-form mixed-modal generation
evaluation, where either the prompt or outputs contain mixed sequences of both
images and text. Chameleon marks a significant step forward in a unified
modeling of full multimodal documents.

Chameleon 是一种早期融合的基于令牌的混合模态模型系列，能够理解和生成任意顺序的图像和文本。它在图像问答、图像描述、文本生成、图像生成和长格式混合模态生成等任务上展现了广泛和通用的能力，包括在图像描述任务中达到了最先进的性能水平，同时在纯文本任务中超越了 Llama-2，与 Mixtral 8x7B 和 Gemini-Pro 等模型竞争，并且能够生成非平凡的图像，所有这些能力都集成在一个模型中。它还在全新的长格式混合模态生成评估中，通过人类评判匹配或超越了更大模型，包括 Gemini Pro 和 GPT-4V，该评估中的提示或输出包含混合的图像和文本序列。Chameleon 在统一建模完整多模态文档方面迈出了重要的一步。

变色龙：混合模式早期融合基础模型

Chameleon: Mixed-Modal Early-Fusion Foundation Models

A Retrieval-Augmented Language Model (RALM) augments a generative language
model by retrieving context-specific knowledge from an external database. This
strategy facilitates impressive text generation quality even with smaller
models, thus reducing orders of magnitude of computational demands. However,
RALMs introduce unique system design challenges due to (a) the diverse workload
characteristics between LM inference and retrieval and (b) the various system
requirements and bottlenecks for different RALM configurations such as model
sizes, database sizes, and retrieval frequencies. We propose Chameleon, a
heterogeneous accelerator system that integrates both LM and retrieval
accelerators in a disaggregated architecture. The heterogeneity ensures
efficient acceleration of both LM inference and retrieval, while the
accelerator disaggregation enables the system to independently scale both types
of accelerators to fulfill diverse RALM requirements. Our Chameleon prototype
implements retrieval accelerators on FPGAs and assigns LM inference to GPUs,
with a CPU server orchestrating these accelerators over the network. Compared
to CPU-based and CPU-GPU vector search systems, Chameleon achieves up to 23.72x
speedup and 26.2x energy efficiency. Evaluated on various RALMs, Chameleon
exhibits up to 2.16x reduction in latency and 3.18x speedup in throughput
compared to the hybrid CPU-GPU architecture. These promising results pave the
way for bringing accelerator heterogeneity and disaggregation into future RALM
systems.

创建了一种名为 Chameleon 的异构加速器系统，利用分体架构整合了语言模型和信息检索加速器，实现了对于不同的 Retrieval-Augmented Language Model 系统需求的高效加速，并在性能上取得了显著的提升。

变色龙：一种用于检索增强语言模型的异构和解聚加速器系统

Chameleon: a Heterogeneous and Disaggregated Accelerator System for  Retrieval-Augmented Language Models

Large language models (LLMs) have achieved remarkable progress in various
natural language processing tasks with emergent abilities. However, they face
inherent limitations, such as an inability to access up-to-date information,
utilize external tools, or perform precise mathematical reasoning. In this
paper, we introduce Chameleon, a plug-and-play compositional reasoning
framework that augments LLMs to help address these challenges. Chameleon
synthesizes programs to compose various tools, including LLM models,
off-the-shelf vision models, web search engines, Python functions, and
rule-based modules tailored to user interests. Built on top of an LLM as a
natural language planner, Chameleon infers the appropriate sequence of tools to
compose and execute in order to generate a final response. We showcase the
adaptability and effectiveness of Chameleon on two tasks: ScienceQA and TabMWP.
Notably, Chameleon with GPT-4 achieves an 86.54% accuracy on ScienceQA,
significantly improving upon the best published few-shot model by 11.37%; using
GPT-4 as the underlying LLM, Chameleon achieves a 17.8% increase over the
state-of-the-art model, leading to a 98.78% overall accuracy on TabMWP. Further
studies suggest that using GPT-4 as a planner exhibits more consistent and
rational tool selection and is able to infer potential constraints given the
instructions, compared to other LLMs like ChatGPT.

本文提出了 Chameleon 框架，该框架通过在 Large language models（LLMs）上加上合成程序的方式来提高 LLMs 的能力，特别是在处理需要精确数学推理、使用外部工具或获取最新信息的任务时。作者通过使用科学问题答案（ScienceQA）和网页表格解析（TabMWP）两个任务来展示 Chameleon 的效率和适应性，结果表明 Chameleon 可以在这些任务上显著提高模型的准确率，并且使用 GPT-4 作为规划器可以更加稳定和合理地选择工具，并推断给定指令的潜在约束。