In-context learning (ICL) facilitates Large Language Models (LLMs) exhibiting
emergent ability on downstream tasks without updating billions of parameters.
However, in the area of multi-modal Large Language Models (MLLMs), two problems
hinder the application of multi-modal ICL: (1) Most primary MLLMs are only
trained on single-image datasets, making them unable to read multi-modal
demonstrations. (2) With the demonstrations increasing, thousands of visual
tokens highly challenge hardware and degrade ICL performance. During
preliminary explorations, we discovered that the inner LLM tends to focus more
on the linguistic modality within multi-modal demonstrations to generate
responses. Therefore, we propose a general and light-weighted framework
\textbf{AIM} to tackle the mentioned problems through \textbf{A}ggregating
\textbf{I}mage information of \textbf{M}ultimodal demonstrations to the dense
latent space of the corresponding linguistic part. Specifically, AIM first uses
the frozen backbone MLLM to read each image-text demonstration and extracts the
vector representations on top of the text. These vectors naturally fuse the
information of the image-text pair, and AIM transforms them into fused virtual
tokens acceptable for the inner LLM via a trainable projection layer.
Ultimately, these fused tokens function as variants of multi-modal
demonstrations, fed into the MLLM to direct its response to the current query
as usual. Because these fused tokens stem from the textual component of the
image-text pair, a multi-modal demonstration is nearly reduced to a pure
textual demonstration, thus seamlessly applying to any MLLMs. With its de facto
MLLM frozen, AIM is parameter-efficient and we train it on public multi-modal
web corpora which have nothing to do with downstream test tasks.

通过聚合多模态演示的图像信息到相应的语言部分的密集潜在空间，我们提出了一种称为 AIM 的通用轻量级框架来解决多模态 ICL 的两个问题。

让任何多模态大型语言模型都能高效地进行上下文学习

AIM: Let Any Multi-modal Large Language Models Embrace Efficient  In-Context Learning

This paper introduces AIM, a collection of vision models pre-trained with an
autoregressive objective. These models are inspired by their textual
counterparts, i.e., Large Language Models (LLMs), and exhibit similar scaling
properties. Specifically, we highlight two key findings: (1) the performance of
the visual features scale with both the model capacity and the quantity of
data, (2) the value of the objective function correlates with the performance
of the model on downstream tasks. We illustrate the practical implication of
these findings by pre-training a 7 billion parameter AIM on 2 billion images,
that achieves 84.0% on ImageNet-1k with a frozen trunk. Interestingly, even at
this scale, we observe no sign of saturation in performance, suggesting that
AIM potentially represents a new frontier for training large-scale vision
models. The pre-training of AIM is similar to the pre-training of LLMs, and
does not require any image-specific strategy to stabilize the training at
scale.

这篇论文介绍了 AIM，一种根据自回归目标预训练的视觉模型集合。这些模型受到了文本模型的启发，并展现了类似的规模扩展性。重点提到两个关键发现：（1）视觉特征的性能随模型容量和数据量的增加而提高，（2）目标函数的价值与模型在下游任务上的性能相关。通过预先训练一架 70 亿参数、使用 20 亿图像的 AIM，在 ImageNet-1k 上使用冻结的主干获得了 84.0% 的准确率。有趣的是，即使在这样的规模下，我们观察不到性能饱和的迹象，这表明 AIM 可能代表了训练大规模视觉模型的一个新的前沿。AIM 的预训练类似于 LLM 的预训练，并不需要任何图像特定的策略来稳定规模化训练。