In-context learning (ICL) exhibits dual operating modes: task learning, i.e.,
acquiring a new skill from in-context samples, and task retrieval, i.e.,
locating and activating a relevant pretrained skill. Recent theoretical work
investigates various mathematical models to analyze ICL, but existing models
explain only one operating mode at a time. We introduce a probabilistic model,
with which one can explain the dual operating modes of ICL simultaneously.
Focusing on in-context learning of linear functions, we extend existing models
for pretraining data by introducing multiple task groups and task-dependent
input distributions. We then analyze the behavior of the optimally pretrained
model under the squared loss, i.e., the MMSE estimator of the label given
in-context examples. Regarding pretraining task distribution as prior and
in-context examples as the observation, we derive the closed-form expression of
the task posterior distribution. With the closed-form expression, we obtain a
quantitative understanding of the two operating modes of ICL. Furthermore, we
shed light on an unexplained phenomenon observed in practice: under certain
settings, the ICL risk initially increases and then decreases with more
in-context examples. Our model offers a plausible explanation for this "early
ascent" phenomenon: a limited number of in-context samples may lead to the
retrieval of an incorrect skill, thereby increasing the risk, which will
eventually diminish as task learning takes effect with more in-context samples.
We also theoretically analyze ICL with biased labels, e.g., zero-shot ICL,
where in-context examples are assigned random labels. Lastly, we validate our
findings and predictions via experiments involving Transformers and large
language models.

通过引入概率模型，我们对上下文学习的双重工作模式进行了解释，并分析了线性函数的上下文学习行为，展示了一种可能的解释，即通过更多的上下文示例，任务学习将产生作用并减少风险。

上下文学习的双重操作模式

Dual Operating Modes of In-Context Learning

In this paper, we propose a novel sparse signal recovery algorithm called
Trainable ISTA (TISTA). The proposed algorithm consists of two estimation units
such as a linear estimation unit and a minimum mean squared error (MMSE)
estimator-based shrinkage unit. The estimated error variance required in the
MMSE shrinkage unit is precisely estimated from a tentative estimate of the
original signal. The remarkable feature of the proposed scheme is that TISTA
includes adjustable variables controlling a step size and the error variance
for the MMSE shrinkage. The variables are adjusted by standard deep learning
techniques. The number of trainable variables of TISTA is equal to the number
of iteration rounds and it is much smaller than that of known learnable sparse
signal recovery algorithms. This feature leads to highly stable and fast
training processes of TISTA. Computer experiments show that TISTA is applicable
to various classes of sensing matrices such as Gaussian matrices, binary
matrices and matrices with large condition numbers. Numerical results also
demonstrate that TISTA shows significantly faster convergence than those of AMP
and LISTA in many cases.

本文提出了一种名为 TISTA 的新型稀疏信号恢复算法，采用了可调整参数控制步长和误差方差的最小均方误差收缩单元，并通过标准深度学习技术调整这些变量，实验表明，TISTA 适用于各种传感矩阵，并且收敛速度明显快于 AMP 和 LISTA。

可训练 ISTA 用于稀疏信号恢复

Trainable ISTA for Sparse Signal Recovery

We present a method for estimating conditionally Gaussian random vectors with
random covariance matrices, which uses techniques from the field of machine
learning. Such models are typical in communication systems, where the
covariance matrix of the channel vector depends on random parameters, e.g.,
angles of propagation paths. If the covariance matrices exhibit certain
Toeplitz and shift-invariance structures, the complexity of the MMSE channel
estimator can be reduced to O(M log M) floating point operations, where M is
the channel dimension. While in the absence of structure the complexity is much
higher, we obtain a similarly efficient (but suboptimal) estimator by using the
MMSE estimator of the structured model as a blueprint for the architecture of a
neural network. This network learns the MMSE estimator for the unstructured
model, but only within the given class of estimators that contains the MMSE
estimator for the structured model. Numerical simulations with typical spatial
channel models demonstrate the generalization properties of the chosen class of
estimators to realistic channel models.

本文介绍了一种使用机器学习技术来估计具有随机协方差矩阵的条件高斯随机向量的方法。这种模型在通信系统中非常常见，通过使用建立在结构化模型上 MMSE 估计器的神经网络蓝图，可以在给定精度下获得相似有效的解决方案，同时具备对理论模型之外的实际环境的一些基础的适应性。