As model context lengths continue to increase, the number of demonstrations
that can be provided in-context approaches the size of entire training
datasets. We study the behavior of in-context learning (ICL) at this extreme
scale on multiple datasets and models. We show that, for many datasets with
large label spaces, performance continues to increase with hundreds or
thousands of demonstrations. We contrast this with example retrieval and
finetuning: example retrieval shows excellent performance at low context
lengths but has diminished gains with more demonstrations; finetuning is more
data hungry than ICL but can sometimes exceed long-context ICL performance with
additional data. We use this ICL setting as a testbed to study several
properties of both in-context learning and long-context models. We show that
long-context ICL is less sensitive to random input shuffling than short-context
ICL, that grouping of same-label examples can negatively impact performance,
and that the performance boosts we see do not arise from cumulative gain from
encoding many examples together. We conclude that although long-context ICL can
be surprisingly effective, most of this gain comes from attending back to
similar examples rather than task learning.

当模型的上下文长度不断增加时，可以提供的演示数量接近整个训练数据集的规模。在多个数据集和模型上，我们研究了在这种极端规模下的上下文学习（ICL）的行为。我们展示了对于许多具有大型标签空间的数据集，性能在数百或数千个演示中仍在增加。与示例检索和微调相比，示例检索在较短的上下文长度下表现出色，但随着更多的演示，增益减弱；微调比 ICL 更依赖数据，但有时可以通过额外数据超过长上下文 ICL 的性能。我们在 ICL 设置中使用这个作为试验平台来研究上下文学习和长上下文模型的几个特性。我们展示了长上下文 ICL 比短上下文 ICL 对于随机输入乱序不太敏感，相同标签示例的分组可能对性能产生负面影响，并且我们观察到的性能提升并非来自一起编码多个示例的累积增益。我们得出结论，尽管长上下文 ICL 可能出人意料地有效，但其中大部分增益来自于重新关注类似示例而非任务学习。