In-Context Learning (ICL) has emerged as a pivotal capability of auto-regressive large language models, yet it is hindered by a notable sensitivity to the ordering of context examples regardless of their mutual independence. To address this issue, recent studies have introduced several variant algorithms of ICL that achieve permutation invariance. However, many of these do not exhibit comparable performance with the standard auto-regressive ICL algorithm. In this work, we identify two crucial elements in the design of an invariant ICL algorithm: information non-leakage and context interdependence, which are not simultaneously achieved by any of the existing methods. These investigations lead us to the proposed Invariant ICL (InvICL), a methodology designed to achieve invariance in ICL while ensuring the two properties. Empirically, our findings reveal that InvICL surpasses previous models, both invariant and non-invariant, in most benchmark datasets, showcasing superior generalization capabilities across varying input lengths. Code is available at https://github.com/PKU-ML/InvICL.

本文针对自回归大型语言模型中出现的上下文学习（ICL）对上下文示例顺序敏感的问题，提出了一种新的方法。通过识别信息非泄漏和上下文相互依赖性这两个关键要素，本文开发了不变ICL（InvICL）方法。实验证明，InvICL在大多数基准数据集上超越了现有模型，展现了更强的泛化能力。

重新思考上下文学习中的不变性