In-context learning (ICL) facilitates Large Language Models (LLMs) exhibiting
emergent ability on downstream tasks without updating billions of parameters.
However, in the area of multi-modal Large Language Models (MLLMs), two problems
hinder the application of multi-modal ICL: (1) Most primary MLLMs are only
trained on single-image datasets, making them unable to read multi-modal
demonstrations. (2) With the demonstrations increasing, thousands of visual
tokens highly challenge hardware and degrade ICL performance. During
preliminary explorations, we discovered that the inner LLM tends to focus more
on the linguistic modality within multi-modal demonstrations to generate
responses. Therefore, we propose a general and light-weighted framework
\textbf{AIM} to tackle the mentioned problems through \textbf{A}ggregating
\textbf{I}mage information of \textbf{M}ultimodal demonstrations to the dense
latent space of the corresponding linguistic part. Specifically, AIM first uses
the frozen backbone MLLM to read each image-text demonstration and extracts the
vector representations on top of the text. These vectors naturally fuse the
information of the image-text pair, and AIM transforms them into fused virtual
tokens acceptable for the inner LLM via a trainable projection layer.
Ultimately, these fused tokens function as variants of multi-modal
demonstrations, fed into the MLLM to direct its response to the current query
as usual. Because these fused tokens stem from the textual component of the
image-text pair, a multi-modal demonstration is nearly reduced to a pure
textual demonstration, thus seamlessly applying to any MLLMs. With its de facto
MLLM frozen, AIM is parameter-efficient and we train it on public multi-modal
web corpora which have nothing to do with downstream test tasks.

通过聚合多模态演示的图像信息到相应的语言部分的密集潜在空间，我们提出了一种称为 AIM 的通用轻量级框架来解决多模态 ICL 的两个问题。

让任何多模态大型语言模型都能高效地进行上下文学习

AIM: Let Any Multi-modal Large Language Models Embrace Efficient  In-Context Learning

Learning skills by imitation is a promising concept for the intuitive
teaching of robots. A common way to learn such skills is to learn a parametric
model by maximizing the likelihood given the demonstrations. Yet, human
demonstrations are often multi-modal, i.e., the same task is solved in multiple
ways which is a major challenge for most imitation learning methods that are
based on such a maximum likelihood (ML) objective. The ML objective forces the
model to cover all data, it prevents specialization in the context space and
can cause mode-averaging in the behavior space, leading to suboptimal or
potentially catastrophic behavior. Here, we alleviate those issues by
introducing a curriculum using a weight for each data point, allowing the model
to specialize on data it can represent while incentivizing it to cover as much
data as possible by an entropy bonus. We extend our algorithm to a Mixture of
(linear) Experts (MoE) such that the single components can specialize on local
context regions, while the MoE covers all data points. We evaluate our approach
in complex simulated and real robot control tasks and show it learns from
versatile human demonstrations and significantly outperforms current SOTA
methods. A reference implementation can be found at
this https URL

本研究提出一种基于课程学习和 MoE 的模仿学习方法，用于从人类多样化的演示中教授机器人控制技能，并在复杂机器人控制任务中显著优于当前最先进的方法。

基于课程的通用技能模仿

Curriculum-Based Imitation of Versatile Skills

We address the problem of imitation learning with multi-modal demonstrations.
Instead of attempting to learn all modes, we argue that in many tasks it is
sufficient to imitate any one of them. We show that the state-of-the-art
methods such as GAIL and behavior cloning, due to their choice of loss
function, often incorrectly interpolate between such modes. Our key insight is
to minimize the right divergence between the learner and the expert
state-action distributions, namely the reverse KL divergence or I-projection.
We propose a general imitation learning framework for estimating and minimizing
any f-Divergence. By plugging in different divergences, we are able to recover
existing algorithms such as Behavior Cloning (Kullback-Leibler), GAIL (Jensen
Shannon) and Dagger (Total Variation). Empirical results show that our
approximate I-projection technique is able to imitate multi-modal behaviors
more reliably than GAIL and behavior cloning.

本文提出了一种使用多模演示的模仿学习方法，针对现有方法中插值错误的问题，采用与专家状态 - 行动分布的正向 KL 散度相对应的反向 KL 散度，即 I-projection，作为不同 f - 散度估计和最小化的框架，并得出了比 GAIL 和行为克隆更加可靠的多模行为近似 I-projection 方法。