In-Context Learning (ICL) is an important paradigm for adapting Large
Language Models (LLMs) to downstream tasks through a few demonstrations.
Despite the great success of ICL, the limitation of the demonstration number
may lead to demonstration bias, i.e. the input-label mapping induced by LLMs
misunderstands the task's essence. Inspired by human experience, we attempt to
mitigate such bias through the perspective of the inter-demonstration
relationship. Specifically, we construct Comparable Demonstrations (CDs) by
minimally editing the texts to flip the corresponding labels, in order to
highlight the task's essence and eliminate potential spurious correlations
through the inter-demonstration comparison. Through a series of experiments on
CDs, we find that (1) demonstration bias does exist in LLMs, and CDs can
significantly reduce such bias; (2) CDs exhibit good performance in ICL,
especially in out-of-distribution scenarios. In summary, this study explores
the ICL mechanisms from a novel perspective, providing a deeper insight into
the demonstration selection strategy for ICL.

该研究从一种新的视角探索了 In-Context Learning (ICL) 机制，通过构建 Comparable Demonstrations (CDs) 来减少演示偏差，从而提供了对 ICL 演示选择策略的更深入理解。