The integration of visual inputs with large language models (LLMs) has led to
remarkable advancements in multi-modal capabilities, giving rise to visual
large language models (VLLMs). However, effectively harnessing VLLMs for
intricate visual perception tasks remains a challenge. In this paper, we
present a novel end-to-end framework named PerceptionGPT, which efficiently and
effectively equips the VLLMs with visual perception abilities by leveraging the
representation power of LLMs' token embedding. Our proposed method treats the
token embedding of the LLM as the carrier of spatial information, then leverage
lightweight visual task encoders and decoders to perform visual perception
tasks (e.g., detection, segmentation). Our approach significantly alleviates
the training difficulty suffered by previous approaches that formulate the
visual outputs as discrete tokens, and enables achieving superior performance
with fewer trainable parameters, less training data and shorted training time.
Moreover, as only one token embedding is required to decode the visual outputs,
the resulting sequence length during inference is significantly reduced.
Consequently, our approach enables accurate and flexible representations,
seamless integration of visual perception tasks, and efficient handling of a
multiple of visual outputs. We validate the effectiveness and efficiency of our
approach through extensive experiments. The results demonstrate significant
improvements over previous methods with much fewer trainable parameters and GPU
hours, which facilitates future research in enabling LLMs with visual
perception abilities.

本文提出了一种名为 PerceptionGPT 的新型端到端框架，通过利用 LLMs 的 token 嵌入的表示能力，高效有效地赋予 VLLMs 视觉感知能力。该方法以 LLMs 的 token 嵌入作为空间信息的携带者，利用轻量级的视觉任务编码器和解码器执行视觉感知任务（如检测、分割），有效缓解了以往将视觉输出离散化为 token 的训练困难，并且能够在更少的可训练参数、较少的训练数据和较短的训练时间内实现更优越的性能。此外，由于推理过程中只需要一个 token 嵌入来解码视觉输出，结果序列长度可大幅减少。因此，该方法能够实现准确灵活的表示，无缝集成视觉感知任务，并高效处理多个视觉输出，通过广泛的实验证实了该方法的有效性和效率，结果表明，在更少的可训练参数和 GPU 时间的情况下取得了显著的改进，为未来赋予 LLMs 视觉感知能力的研究提供了便利。