Large language models (LLMs) are able to solve various tasks with only a few
demonstrations utilizing their in-context learning (ICL) abilities. However,
LLMs often rely on their pre-trained semantic priors of demonstrations rather
than on the input-label relationships to proceed with ICL prediction. In this
work, we term this phenomenon as the `Demonstration Shortcut'. While previous
works have primarily focused on improving ICL prediction results for predefined
tasks, we aim to rectify the Demonstration Shortcut, thereby enabling the LLM
to effectively learn new input-label relationships from demonstrations. To
achieve this, we introduce In-Context Calibration, a demonstration-aware
calibration method. We evaluate the effectiveness of the proposed method in two
settings: (1) the Original ICL Task using the standard label space and (2) the
Task Learning setting, where the label space is replaced with semantically
unrelated tokens. In both settings, In-Context Calibration demonstrates
substantial improvements, with results generalized across three LLM families
(OPT, GPT, and Llama2) under various configurations.

大型语言模型通过它们的上下文学习能力能够只使用几个示例来解决各种任务。然而，LLMs 通常依赖于它们预先训练的演示案例的语义先验知识而不是依靠输入 - 标签关系来进行上下文学习预测。本研究有意解决这个现象，引入了一种称为 “演示捷径” 的现象。通过介绍一种演示感知的校准方法 ——In-Context Calibration，我们旨在使 LLM 能够有效地从示例中学习新的输入 - 标签关系。我们在两个场景下评估了所提出方法的有效性：（1）使用标准标签空间的原始上下文学习任务和（2）任务学习设置，其中标签空间由语义上不相关的令牌替代。在这两个场景中，In-Context Calibration 表现出显著的改进效果，并在多个 LLM 家族（OPT，GPT 和 Llama2）的各种配置下推广得到了结果。