In recent years, advances in the large-scale pretraining of language and
text-to-image models have revolutionized the field of machine learning. Yet,
integrating these two modalities into a single, robust model capable of
generating seamless multimodal outputs remains a significant challenge. To
address this gap, we present the Joint Autoregressive Mixture (JAM) framework,
a modular approach that systematically fuses existing text and image generation
models. We also introduce a specialized, data-efficient instruction-tuning
strategy, tailored for mixed-modal generation tasks. Our final instruct-tuned
model demonstrates unparalleled performance in generating high-quality
multimodal outputs and represents the first model explicitly designed for this
purpose.

近年来，语言和文本到图像模型的大规模预训练的进步已经革新了机器学习领域。然而，将这两种模态集成到一个能够生成无缝多模态输出的强大模型仍然是一个重大挑战。为了解决这个问题，我们提出了联合自回归混合（JAM）框架，这是一种模块化的方法，系统地融合了现有的文本和图像生成模型。我们还引入了一种专门的、高效的数据调优策略，为混合模态生成任务量身定制。我们最终调优的模型在生成高质量多模态输出方面表现出了无与伦比的性能，并且代表了首个明确设计用于此目的的模型。