Despite recent explosion in research interests, in-context learning and the precise impact of the quality of demonstrations remain elusive. While, based on current literature, it is expected that in-context learning shares a similar mechanism to supervised learning, Min et al. (2022) recently reported that, surprisingly, input-label correspondence is less important than other aspects of prompt demonstrations. Inspired by this counter-intuitive observation, we re-examine the importance of ground truth labels on in-context learning from diverse and statistical points of view. With the aid of the newly introduced metrics, i.e., Ground-truth Label Effect Ratio (GLER), demo-gain, and label sensitivity, we find that the impact of the correct input-label matching can vary according to different configurations. Expanding upon the previous key finding on the role of demonstrations, the complementary and contrastive results suggest that one might need to take more care when estimating the impact of each component in in-context learning demonstrations.

我们基于`Label-Correctness Sensitivity`和`Ground-truth Label Effect Ratio`这两个新颖的度量标准，重新审视了在上下文学习中使用示范的重要性，发现正确的输入-标签映射对上下文学习的性能有着不同的影响，取决于实验设置等关键组成部分。我们还确定了`提示模板的详细程度`和`语言模型大小`等因素是实现更强的抗噪声上下文学习的关键因素。

真实标签很重要：深入探究输入标签演示