We present Emu, a Transformer-based multimodal foundation model, which can
seamlessly generate images and texts in multimodal context. This omnivore model
can take in any single-modality or multimodal data input indiscriminately
(e.g., interleaved image, text and video) through a one-model-for-all
autoregressive training process. First, visual signals are encoded into
embeddings, and together with text tokens form an interleaved input sequence.
Emu is then end-to-end trained with a unified objective of classifying the next
text token or regressing the next visual embedding in the multimodal sequence.
This versatile multimodality empowers the exploration of diverse pretraining
data sources at scale, such as videos with interleaved frames and text,
webpages with interleaved images and text, as well as web-scale image-text
pairs and video-text pairs. Emu can serve as a generalist multimodal interface
for both image-to-text and text-to-image tasks, and supports in-context image
and text generation. Across a broad range of zero-shot/few-shot tasks including
image captioning, visual question answering, video question answering and
text-to-image generation, Emu demonstrates superb performance compared to
state-of-the-art large multimodal models. Extended capabilities such as
multimodal assistants via instruction tuning are also demonstrated with
impressive performance.

本文介绍了 Emu—— 一种基于 Transformer 的多模态基础模型，它可以在多模态语境中无缝生成图像和文本。该模型可通过单一模型进行全自回归训练，将任何单模态或多模态数据输入混合 (例如，交错的图像、文本和视频)，并表现得十分出色。

多模态生成预训练

Generative Pretraining in Multimodality

We present Emu, a system that semantically enhances multilingual sentence
embeddings. Our framework fine-tunes pre-trained multilingual sentence
embeddings using two main components: a semantic classifier and a language
discriminator. The semantic classifier improves the semantic similarity of
related sentences, whereas the language discriminator enhances the
multilinguality of the embeddings via multilingual adversarial training. Our
experimental results based on several language pairs show that our specialized
embeddings outperform the state-of-the-art multilingual sentence embedding
model on the task of cross-lingual intent classification using only monolingual
labeled data.

Emu 是一个可在多语言中加强句子语义的系统，其利用语义分类器和语言鉴别器两大主要组件对预训练的多语言句子嵌入进行微调，以提高相关句子的语义相似度和通过多语言对抗式训练来增强多语言性能，并经多种语言对的实验结果表明，在仅使用单语标记数据的情况下，我们专门的嵌入模型在跨语言意图分类任务上优于最先进的多语言句子嵌入模型。