Autoregressively trained transformers have brought a profound revolution to
the world, especially with their in-context learning (ICL) ability to address
downstream tasks. Recently, several studies suggest that transformers learn a
mesa-optimizer during autoregressive (AR) pretraining to implement ICL. Namely,
the forward pass of the trained transformer is equivalent to optimizing an
inner objective function in-context. However, whether the practical non-convex
training dynamics will converge to the ideal mesa-optimizer is still unclear.
Towards filling this gap, we investigate the non-convex dynamics of a one-layer
linear causal self-attention model autoregressively trained by gradient flow,
where the sequences are generated by an AR process $x_{t+1} = W x_t$. First,
under a certain condition of data distribution, we prove that an
autoregressively trained transformer learns $W$ by implementing one step of
gradient descent to minimize an ordinary least squares (OLS) problem
in-context. It then applies the learned $\widehat{W}$ for next-token
prediction, thereby verifying the mesa-optimization hypothesis. Next, under the
same data conditions, we explore the capability limitations of the obtained
mesa-optimizer. We show that a stronger assumption related to the moments of
data is the sufficient and necessary condition that the learned mesa-optimizer
recovers the distribution. Besides, we conduct exploratory analyses beyond the
first data condition and prove that generally, the trained transformer will not
perform vanilla gradient descent for the OLS problem. Finally, our simulation
results verify the theoretical results.

最近的研究表明，自回归训练的 Transformer 学习了一个内在的优化器，通过正向传播来优化一个内部目标函数，我们探索了一个通过梯度流进行自回归训练的一层线性因果自注意模型的非凸动力学，以填补该领域的知识空白，并验证了该模型能够实现内在学习能力和优化器假设。