Transformer-based large language models have displayed impressive in-context
learning capabilities, where a pre-trained model can handle new tasks without
fine-tuning by simply augmenting the query with some input-output examples from
that task. Despite the empirical success, the mechanics of how to train a
Transformer to achieve ICL and the corresponding ICL capacity is mostly elusive
due to the technical challenges of analyzing the nonconvex training problems
resulting from the nonlinear self-attention and nonlinear activation in
Transformers. To the best of our knowledge, this paper provides the first
theoretical analysis of the training dynamics of Transformers with nonlinear
self-attention and nonlinear MLP, together with the ICL generalization
capability of the resulting model. Focusing on a group of binary classification
tasks, we train Transformers using data from a subset of these tasks and
quantify the impact of various factors on the ICL generalization performance on
the remaining unseen tasks with and without data distribution shifts. We also
analyze how different components in the learned Transformers contribute to the
ICL performance. Furthermore, we provide the first theoretical analysis of how
model pruning affects the ICL performance and prove that proper magnitude-based
pruning can have a minimal impact on ICL while reducing inference costs. These
theoretical findings are justified through numerical experiments.

通过理论分析，我们首次探讨了具有非线性自注意力和非线性 MLP 的 Transformer 模型的训练动态和 ICL 泛化能力，重点关注一组二分类任务，研究了各种因素对 ICL 泛化性能的影响，探讨了不同组件对 ICL 性能的贡献，并首次理论分析了模型修剪对 ICL 性能的影响，证明合适的基于大小的修剪可以在降低推理成本的同时对 ICL 产生最小影响，并通过数值实验验证了这些理论结果。