The ability to learn from context with novel concepts, and deliver
appropriate responses are essential in human conversations. Despite current
Multimodal Large Language Models (MLLMs) and Large Language Models (LLMs) being
trained on mega-scale datasets, recognizing unseen images or understanding
novel concepts in a training-free manner remains a challenge. In-Context
Learning (ICL) explores training-free few-shot learning, where models are
encouraged to ``learn to learn" from limited tasks and generalize to unseen
tasks. In this work, we propose link-context learning (LCL), which emphasizes
"reasoning from cause and effect" to augment the learning capabilities of
MLLMs. LCL goes beyond traditional ICL by explicitly strengthening the causal
relationship between the support set and the query set. By providing
demonstrations with causal links, LCL guides the model to discern not only the
analogy but also the underlying causal associations between data points, which
empowers MLLMs to recognize unseen images and understand novel concepts more
effectively. To facilitate the evaluation of this novel approach, we introduce
the ISEKAI dataset, comprising exclusively of unseen generated image-label
pairs designed for link-context learning. Extensive experiments show that our
LCL-MLLM exhibits strong link-context learning capabilities to novel concepts
over vanilla MLLMs. Code and data will be released at
this https URL

基于多模式大型语言模型和大型语言模型，提出了一种新颖的上下文学习方法（In-Context Learning），通过加强因果关系，引导模型识别类比和数据点间的潜在因果关联，使模型能够识别未见过的图像和理解新概念更有效。

多模式 LLMs 的链接上下文学习

Link-Context Learning for Multimodal LLMs

There is a growing interest in developing automated agents that can work
alongside humans. In addition to completing the assigned task, such an agent
will undoubtedly be expected to behave in a manner that is preferred by the
human. This requires the human to communicate their preferences to the agent.
To achieve this, the current approaches either require the users to specify the
reward function or the preference is interactively learned from queries that
ask the user to compare behavior. The former approach can be challenging if the
internal representation used by the agent is inscrutable to the human while the
latter is unnecessarily cumbersome for the user if their preference can be
specified more easily in symbolic terms. In this work, we propose PRESCA
(PREference Specification through Concept Acquisition), a system that allows
users to specify their preferences in terms of concepts that they understand.
PRESCA maintains a set of such concepts in a shared vocabulary. If the relevant
concept is not in the shared vocabulary, then it is learned. To make learning a
new concept more feedback efficient, PRESCA leverages causal associations
between the target concept and concepts that are already known. In addition, we
use a novel data augmentation approach to further reduce required feedback. We
evaluate PRESCA by using it on a Minecraft environment and show that it can
effectively align the agent with the user's preference.

本文提出了 PRESCA 系统，使用概念学习以及因果关联和数据增强等方法，让用户更轻松地通过概念来指定其对自动化代理的偏好，并在 Minecraft 环境中验证其有效性。