In this work, we introduce Semantic Pyramid AutoEncoder (SPAE) for enabling
frozen LLMs to perform both understanding and generation tasks involving
non-linguistic modalities such as images or videos. SPAE converts between raw
pixels and interpretable lexical tokens (or words) extracted from the LLM's
vocabulary. The resulting tokens capture both the semantic meaning and the
fine-grained details needed for visual reconstruction, effectively translating
the visual content into a language comprehensible to the LLM, and empowering it
to perform a wide array of multimodal tasks. Our approach is validated through
in-context learning experiments with frozen PaLM 2 and GPT 3.5 on a diverse set
of image understanding and generation tasks. Our method marks the first
successful attempt to enable a frozen LLM to generate image content while
surpassing state-of-the-art performance in image understanding tasks, under the
same setting, by over 25%.

本文介绍了语义金字塔自编码器（SPAE），它使得具有非语言模式（如图像或视频）的被冻结的 LLMs 能够执行理解和生成任务。我们的方法通过在冻结的 PaLM 2 和 GPT 3.5 上进行多样化的图像理解和生成任务中上下文学习实验证明，是首次成功尝试使冻结的 LLM 生成图像内容，并在相同的环境中在图像理解任务中表现优于现有技术 25％以上。