In-context learning (ICL) enables large language models (LLMs) to perform new
tasks by using sample-label pairs as demonstrations. However, variations in
demonstrations can lead to significantly different performances. Current
research mainly focuses on selecting demonstration samples, preassuming the
class name to be the label word when creating sample-label pairs. However, the
choice of label words is crucial for ICL performance. In addition, we observe
that using a single class name in demonstration may not yield optimal results.
In this paper, we propose to use multiple label words in one sample-label pair
to enhance ICL performance. Further, we select and order sample-label pairs
based on LLM's output distribution, aiming to optimize the demonstration
examples from both the samples' and labels' perspectives. Evaluation results on
seven classification datasets show that the use of multiple label words,
strategically organized by their selection, order and quantity, improves ICL
performance through diverse label information.

通过使用多个标签词，并根据语言模型的输出分布选择和排序样本 - 标签对，可以提高上下文学习的性能，通过多样的标签信息改善性能。

通过演示中的多标签词改进上下文学习

MICL: Improving In-Context Learning through Multiple-Label Words in  Demonstration

Recently, in-context learning (ICL) on large language models (LLMs) has
received great attention, and this technique can also be applied to
vision-language models (VLMs) built upon LLMs. These VLMs can respond to
queries by conditioning responses on a series of multimodal demonstrations,
which comprise images, queries, and answers. Though ICL has been extensively
studied on LLMs, its research on VLMs remains limited. The inclusion of
additional visual information in the demonstrations motivates the following
research questions: which of the two modalities in the demonstration is more
significant? How can we select effective multimodal demonstrations to enhance
ICL performance? This study investigates the significance of both visual and
language information. Our findings indicate that ICL in VLMs is predominantly
driven by the textual information in the demonstrations whereas the visual
information in the demonstrations barely affects the ICL performance.
Subsequently, we provide an understanding of the findings by analyzing the
model information flow and comparing model inner states given different ICL
settings. Motivated by our analysis, we propose a simple yet effective
approach, termed Mixed Modality In-Context Example Selection (MMICES), which
considers both visual and language modalities when selecting demonstrations and
shows better ICL performance. Extensive experiments are conducted to support
our findings, understanding, and improvement of the ICL performance of VLMs.

通过对视觉 - 语言模型的大规模（LLMs）模型进行背景学习（ICL），本研究发现在 VLMs 中，ICL 主要受到演示的文本信息的驱动，视觉信息对 ICL 性能的影响较小。鉴于该发现，通过分析模型信息流和不同 ICL 设置下的模型内部状态，我们提出了一个简单而有效的方法 MMICES（Mixed Modality In-Context Example Selection），它在选择演示时考虑了视觉和语言两个模态，并显示出更好的 ICL 性能。通过大量实验证实了我们的发现，对 VLMs 的 ICL 性能的理解和改进进行了支持。

理解和优化视觉 - 语言模型中的上下文学习

Understanding and Improving In-Context Learning on Vision-language  Models

While large language models based on the transformer architecture have
demonstrated remarkable in-context learning (ICL) capabilities, understandings
of such capabilities are still in an early stage, where existing theory and
mechanistic understanding focus mostly on simple scenarios such as learning
simple function classes. This paper takes initial steps on understanding ICL in
more complex scenarios, by studying learning with representations. Concretely,
we construct synthetic in-context learning problems with a compositional
structure, where the label depends on the input through a possibly complex but
fixed representation function, composed with a linear function that differs in
each instance. By construction, the optimal ICL algorithm first transforms the
inputs by the representation function, and then performs linear ICL on top of
the transformed dataset. We show theoretically the existence of transformers
that approximately implement such algorithms with mild depth and size.
Empirically, we find trained transformers consistently achieve near-optimal ICL
performance in this setting, and exhibit the desired dissection where lower
layers transforms the dataset and upper layers perform linear ICL. Through
extensive probing and a new pasting experiment, we further reveal several
mechanisms within the trained transformers, such as concrete copying behaviors
on both the inputs and the representations, linear ICL capability of the upper
layers alone, and a post-ICL representation selection mechanism in a harder
mixture setting. These observed mechanisms align well with our theory and may
shed light on how transformers perform ICL in more realistic scenarios.

大型语言模型在转换器架构的基础上展现了卓越的上下文学习能力，本研究旨在深入了解更复杂的情境中的上下文学习，并通过研究表示学习来探索其机制和性能。