Large Language Models (LLMs) have demonstrated impressive performance on
multimodal tasks, without any multimodal finetuning. They are the building
block for Large Multimodal Models, yet, we still lack a proper understanding of
their success. In this work, we expose frozen LLMs to image, video, audio and
text inputs and analyse their internal representation aiming to understand
their generalization beyond textual inputs.
Findings. Perceptual tokens (1) are easily distinguishable from textual ones
inside LLMs, with significantly different representations, and complete
translation to textual tokens does not exist. Yet, (2) both perceptual and
textual tokens activate similar LLM weights. Despite being different, (3)
perceptual and textual tokens are implicitly aligned inside LLMs, we call this
the implicit multimodal alignment (IMA), and argue that this is linked to
architectural design, helping LLMs to generalize. This provide more evidence to
believe that the generalization of LLMs to multimodal inputs is mainly due to
their architecture.
Implications. (1) We find a positive correlation between the implicit
alignment score and the task performance, suggesting that this could act as a
proxy metric for model evaluation and selection. (2) A negative correlation
exists regarding hallucinations, revealing that this problem is mainly due to
misalignment between the internal perceptual and textual representations. (3)
Perceptual tokens change slightly throughout the model, thus, we propose
different approaches to skip computations (e.g. in FFN layers), and
significantly reduce the inference cost. (4) Due to the slowly changing
embeddings across layers, and the high overlap between textual and multimodal
activated weights, we compress LLMs by keeping only 1 subnetwork that works
well across a wide range of multimodal tasks. Paper code:
this https URL

大型语言模型（LLMs）对多模态任务表现出令人印象深刻的性能，然而，我们仍然缺乏对其成功的适当理解。本研究将冻结的 LLMs 公开显示为图像、视频、音频和文本输入，分析其内部表示以了解其在文本输入之外的泛化能力。