Large language models demonstrate an emergent ability to learn a new task
from a small number of input-output demonstrations, referred to as in-context
few-shot learning. However, recent work shows that in such settings, models
mainly learn to mimic the new task distribution, instead of the mechanics of
the new task. We argue that the commonly-used evaluation settings of few-shot
models utilizing a random selection of in-context demonstrations is not able to
disentangle models' ability to learn new skills from demonstrations, as most of
the such-selected demonstrations are not informative for prediction beyond
exposing the new task's input and output distribution.
Therefore, we introduce an evaluation technique that disentangles few-shot
learners' gain from in-context learning by picking the demonstrations sharing a
specific, informative concept with the predicted sample, in addition to the
performance reached by mainly non-informative samples. We find that regardless
of the model size, existing few-shot learners are not able to benefit from
observing such informative concepts in demonstrations. We also find that such
ability may not be obtained trivially by exposing the informative
demonstrations in the training process, leaving the challenge of training true
in-context learners open.

通过挑选与预测样本分享特定信息的示范来区分少样本学习者的能力，结果表明现有的少样本学习者不利用信息，并且暴露了训练真正的上下文学习者的挑战。

上下文中缺少什么？利用信息化演示评估少样学习器

What is Not in the Context? Evaluation of Few-shot Learners with Informative Demonstrations

Existing methods in the Visual Storytelling field often suffer from the
problem of generating general descriptions, while the image contains a lot of
meaningful contents remaining unnoticed. The failure of informative story
generation can be concluded to the model's incompetence of capturing enough
meaningful concepts. The categories of these concepts include entities,
attributes, actions, and events, which are in some cases crucial to grounded
storytelling. To solve this problem, we propose a method to mine the
cross-modal rules to help the model infer these informative concepts given
certain visual input. We first build the multimodal transactions by
concatenating the CNN activations and the word indices. Then we use the
association rule mining algorithm to mine the cross-modal rules, which will be
used for the concept inference. With the help of the cross-modal rules, the
generated stories are more grounded and informative. Besides, our proposed
method holds the advantages of interpretation, expandability, and
transferability, indicating potential for wider application. Finally, we
leverage these concepts in our encoder-decoder framework with the attention
mechanism. We conduct several experiments on the VIsual StoryTelling~(VIST)
dataset, the results of which demonstrate the effectiveness of our approach in
terms of both automatic metrics and human evaluation. Additional experiments
are also conducted showing that our mined cross-modal rules as additional
knowledge helps the model gain better performance when trained on a small
dataset.

本研究提出了一种基于跨模态规则挖掘的视觉叙事方法，通过将 CNN 激活和 Word 指数相结合构建多模态交易，并使用关联规则挖掘算法挖掘跨模态规则，充分捕捉实体、属性、行为和事件等相关概念，将其应用于编码器 - 解码器框架中，得到更具可解释性和可扩展性的故事生成模型，在 VIST 数据集上得到了很好的效果。