In this work, we investigate the potential of a large language model (LLM) to
directly comprehend visual signals without the necessity of fine-tuning on
multi-modal datasets. The foundational concept of our method views an image as
a linguistic entity, and translates it to a set of discrete words derived from
the LLM's vocabulary. To achieve this, we present the Vision-to-Language
Tokenizer, abbreviated as V2T Tokenizer, which transforms an image into a
``foreign language'' with the combined aid of an encoder-decoder, the LLM
vocabulary, and a CLIP model. With this innovative image encoding, the LLM
gains the ability not only for visual comprehension but also for image
denoising and restoration in an auto-regressive fashion-crucially, without any
fine-tuning. We undertake rigorous experiments to validate our method,
encompassing understanding tasks like image recognition, image captioning, and
visual question answering, as well as image denoising tasks like inpainting,
outpainting, deblurring, and shift restoration. Code and models are available
at this https URL

我们研究了大型语言模型（LLM）在没有对多模态数据集进行微调的情况下，直接理解视觉信号的潜力。我们提出了一种视觉到语言的分词器（V2T Tokenizer），通过编码器 - 解码器、LLM 词汇表和 CLIP 模型将图像转换成 “外语”。通过这种创新的图像编码方式，LLM 不仅具备了视觉理解能力，还能以自回归方式进行图像去噪和恢复，关键是不需要任何微调。我们进行了严格的实验证实了我们的方法，包括图像识别、图像字幕、视觉问题回答等理解任务，以及修复任务，如修复、扩展、去模糊和位移复原。