Many recent language models (LMs) are capable of in-context learning (ICL),
manifested in the LMs' ability to perform a new task solely from
natural-language instruction. Previous work curating in-context learners
assumes that ICL emerges from a vast over-parametrization or the scale of
multi-task training. However, recent theoretical work attributes the ICL
ability to concept-dependent training data and creates functional in-context
learners even in small-scale, synthetic settings.
In this work, we practically explore this newly identified axis of ICL
quality. We propose Concept-aware Training (CoAT), a framework for constructing
training scenarios that make it beneficial for the LM to learn to utilize the
analogical reasoning concepts from demonstrations. We find that by using CoAT,
pre-trained transformers can learn to better utilise new latent concepts from
demonstrations and that such ability makes ICL more robust to the functional
deficiencies of the previous models. Finally, we show that concept-aware
in-context learning is more effective for a majority of new tasks when compared
to traditional instruction tuning, resulting in a performance comparable to the
previous in-context learners using magnitudes of more training data.

通过概念感知训练框架构建训练场景，提高预训练变换器在上下文学习过程中利用新的潜在概念的能力，并使这种能力使得上下文学习对先前模型的功能性缺陷更具鲁棒性，相对于传统指令优化，在大多数新任务上具有更好的效果，并且训练数据规模相当的情况下，表现与先前的上下文学习模型相当。

概念感知数据构建改善语言模型的上下文学习

Concept-aware Data Construction Improves In-context Learning of Language  Models

Many recent language models (LMs) of Transformers family exhibit so-called
in-context learning (ICL) ability, manifested in the LMs' ability to modulate
their function by a task described in a natural language input. Previous work
curating these models assumes that ICL emerges from vast over-parametrization
or the scale of multi-task training. However, a complementary branch of recent
theoretical work attributes ICL emergence to specific properties of training
data and creates functional in-context learners in small-scale, synthetic
settings.
Inspired by recent findings on data properties driving the emergence of ICL,
we propose a method to create LMs able to better utilize the in-context
information, by constructing training scenarios where it is beneficial for the
LM to capture the analogical reasoning concepts. We measure that data sampling
of Concept-aware Training (CoAT) consistently improves models' reasoning
ability. As a result, the in-context learners trained with CoAT on only two
datasets of a single (QA) task perform comparably to larger models trained on
1600+ tasks.

文中介绍了一种名为 CoAT 的优化方法，利用模拟训练数据来帮助语言模型更好的利用其背景知识。使用 CoAT 训练的 In-context learners 性能表现良好，达到了在多任务训练中更大规模模型的性能水平。