The integration of visual inputs with large language models (LLMs) has led to
remarkable advancements in multi-modal capabilities, giving rise to visual
large language models (VLLMs). However, effectively harnessing VLLMs for
intricate visual perception tasks remains a challenge. In this paper, we
present a novel end-to-end framework named PerceptionGPT, which efficiently and
effectively equips the VLLMs with visual perception abilities by leveraging the
representation power of LLMs' token embedding. Our proposed method treats the
token embedding of the LLM as the carrier of spatial information, then leverage
lightweight visual task encoders and decoders to perform visual perception
tasks (e.g., detection, segmentation). Our approach significantly alleviates
the training difficulty suffered by previous approaches that formulate the
visual outputs as discrete tokens, and enables achieving superior performance
with fewer trainable parameters, less training data and shorted training time.
Moreover, as only one token embedding is required to decode the visual outputs,
the resulting sequence length during inference is significantly reduced.
Consequently, our approach enables accurate and flexible representations,
seamless integration of visual perception tasks, and efficient handling of a
multiple of visual outputs. We validate the effectiveness and efficiency of our
approach through extensive experiments. The results demonstrate significant
improvements over previous methods with much fewer trainable parameters and GPU
hours, which facilitates future research in enabling LLMs with visual
perception abilities.

本文提出了一种名为 PerceptionGPT 的新型端到端框架，通过利用 LLMs 的 token 嵌入的表示能力，高效有效地赋予 VLLMs 视觉感知能力。该方法以 LLMs 的 token 嵌入作为空间信息的携带者，利用轻量级的视觉任务编码器和解码器执行视觉感知任务（如检测、分割），有效缓解了以往将视觉输出离散化为 token 的训练困难，并且能够在更少的可训练参数、较少的训练数据和较短的训练时间内实现更优越的性能。此外，由于推理过程中只需要一个 token 嵌入来解码视觉输出，结果序列长度可大幅减少。因此，该方法能够实现准确灵活的表示，无缝集成视觉感知任务，并高效处理多个视觉输出，通过广泛的实验证实了该方法的有效性和效率，结果表明，在更少的可训练参数和 GPU 时间的情况下取得了显著的改进，为未来赋予 LLMs 视觉感知能力的研究提供了便利。

PerceptionGPT: 将视觉感知有效融合到 LLM 中

PerceptionGPT: Effectively Fusing Visual Perception into LLM

The ability to understand and work with numbers (numeracy) is critical for
many complex reasoning tasks. Currently, most NLP models treat numbers in text
in the same way as other tokens---they embed them as distributed vectors. Is
this enough to capture numeracy? We begin by investigating the numerical
reasoning capabilities of a state-of-the-art question answering model on the
DROP dataset. We find this model excels on questions that require numerical
reasoning, i.e., it already captures numeracy. To understand how this
capability emerges, we probe token embedding methods (e.g., BERT, GloVe) on
synthetic list maximum, number decoding, and addition tasks. A surprising
degree of numeracy is naturally present in standard embeddings. For example,
GloVe and word2vec accurately encode magnitude for numbers up to 1,000.
Furthermore, character-level embeddings are even more precise---ELMo captures
numeracy the best for all pre-trained methods---but BERT, which uses sub-word
units, is less exact.

探讨了 NLP 模型对数字理解（数值能力）的处理方法，发现现有的模型在 DROP 数据集上表现优秀，并在合成的数字相关任务上检验了 BERT、GloVe 等词向量方法，发现字符级别的嵌入方式更加精确，ELMo pre-training 方法的准确性最好。