In-context learning (ICL) refers to the ability of a model to condition on a
few in-context demonstrations (input-output examples of the underlying task) to
generate the answer for a new query input, without updating parameters. Despite
the impressive ICL ability of LLMs, it has also been found that ICL in LLMs is
sensitive to input demonstrations and limited to short context lengths. To
understand the limitations and principles for successful ICL, we conduct an
investigation with ICL linear regression of transformers. We characterize
several Out-of-Distribution (OOD) cases for ICL inspired by realistic LLM ICL
failures and compare transformers with DeepSet, a simple yet powerful
architecture for ICL. Surprisingly, DeepSet outperforms transformers across a
variety of distribution shifts, implying that preserving permutation invariance
symmetry to input demonstrations is crucial for OOD ICL. The phenomenon
specifies a fundamental requirement by ICL, which we termed as ICL invariance.
Nevertheless, the positional encodings in LLMs will break ICL invariance. To
this end, we further evaluate transformers with identical positional encodings
and find preserving ICL invariance in transformers achieves state-of-the-art
performance across various ICL distribution shifts

深入研究了在上下文学习中的限制和成功原则，并通过比较变压器和 DeepSet 架构以保持重要的离域学习不变性的区别，发现保持离域学习不变性对于成功的下一次时关键的。

位置信息对于不变上下文学习至关重要：简单函数类的案例研究

Positional Information Matters for Invariant In-Context Learning: A Case  Study of Simple Function Classes

Meta-learning, or learning to learn, is a machine learning approach that
utilizes prior learning experiences to expedite the learning process on unseen
tasks. As a data-driven approach, meta-learning requires meta-features that
represent the primary learning tasks or datasets, and are estimated
traditonally as engineered dataset statistics that require expert domain
knowledge tailored for every meta-task. In this paper, first, we propose a
meta-feature extractor called Dataset2Vec that combines the versatility of
engineered dataset meta-features with the expressivity of meta-features learned
by deep neural networks. Primary learning tasks or datasets are represented as
hierarchical sets, i.e., as a set of sets, esp. as a set of predictor/target
pairs, and then a DeepSet architecture is employed to regress meta-features on
them. Second, we propose a novel auxiliary meta-learning task with abundant
data called dataset similarity learning that aims to predict if two batches
stem from the same dataset or different ones. In an experiment on a large-scale
hyperparameter optimization task for 120 UCI datasets with varying schemas as a
meta-learning task, we show that the meta-features of Dataset2Vec outperform
the expert engineered meta-features and thus demonstrate the usefulness of
learned meta-features for datasets with varying schemas for the first time.

本文提出了一种基于深层神经网络的元特征提取器 Dataset2Vec 和基于 DeepSet 结构的元学习方法，在大规模数据集优化任务中将元学习应用于不同模式的数据集，实现了性能优于专家工程方法的元特征学习以及元学习辅助任务。