Large language models demonstrate an emergent ability to learn a new task
from a small number of input-output demonstrations, referred to as in-context
few-shot learning. However, recent work shows that in such settings, models
mainly learn to mimic the new task distribution, instead of the mechanics of
the new task. We argue that the commonly-used evaluation settings of few-shot
models utilizing a random selection of in-context demonstrations is not able to
disentangle models' ability to learn new skills from demonstrations, as most of
the such-selected demonstrations are not informative for prediction beyond
exposing the new task's input and output distribution.
Therefore, we introduce an evaluation technique that disentangles few-shot
learners' gain from in-context learning by picking the demonstrations sharing a
specific, informative concept with the predicted sample, in addition to the
performance reached by mainly non-informative samples. We find that regardless
of the model size, existing few-shot learners are not able to benefit from
observing such informative concepts in demonstrations. We also find that such
ability may not be obtained trivially by exposing the informative
demonstrations in the training process, leaving the challenge of training true
in-context learners open.

通过挑选与预测样本分享特定信息的示范来区分少样本学习者的能力，结果表明现有的少样本学习者不利用信息，并且暴露了训练真正的上下文学习者的挑战。