The rapid advancement of large language models (LLMs) has accelerated the
emergence of in-context learning (ICL) as a cutting-edge approach in the
natural language processing domain. Recently, ICL has been employed in visual
understanding tasks, such as semantic segmentation and image captioning,
yielding promising results. However, existing visual ICL framework can not
enable producing content across multiple modalities, which limits their
potential usage scenarios. To address this issue, we present a new ICL
framework for visual understanding with multi-modal output enabled. First, we
quantize and embed both text and visual prompt into a unified representational
space, structured as interleaved in-context sequences. Then a decoder-only
sparse transformer architecture is employed to perform generative modeling on
them, facilitating in-context learning. Thanks to this design, the model is
capable of handling in-context vision understanding tasks with multimodal
output in a unified pipeline. Experimental results demonstrate that our model
achieves competitive performance compared with specialized models and previous
ICL baselines. Overall, our research takes a further step toward unified
multimodal in-context learning.

我们提出了一种新的具有多模态输出功能的视觉理解的上下文学习框架，通过将文本和视觉提示量化和嵌入到统一的表示空间中，并采用仅具有解码器的稀疏 Transformer 架构在其上执行生成建模。实验结果表明，我们的模型在统一的多模态管线中实现了与专门模型和先前上下文学习基准模型相竞争的性能。总体而言，我们的研究在统一多模态上下文学习方面迈出了进一步的一步。