In this paper, we aim to build the global convergence theory of encoder-only shallow Transformers under a realistic setting from the perspective of architectures, initialization, and scaling under a finite width regime. The difficulty lies in how to tackle the softmax in self-attention mechanism, the core ingredient of Transformer. In particular, we diagnose the scaling scheme, carefully tackle the input/output of softmax, and prove that quadratic overparameterization is sufficient for global convergence of our shallow Transformers under commonly-used He/LeCun initialization in practice. Besides, neural tangent kernel (NTK) based analysis is also given, which facilitates a comprehensive comparison. Our theory demonstrates the separation on the importance of different scaling schemes and initialization. We believe our results can pave the way for a better understanding of modern Transformers, particularly on training dynamics.

该研究建立了关于编码器式浅层Transformer的全局收敛理论，从体系结构、初始化和在有限宽度范围内的缩放的角度进行了现实情景的探究。研究重点在于如何解决Transformer核心组成部分softmax的问题。研究表明，在实践中使用的He/LeCun初始化方式下，我们的浅层Transformer的全局收敛只需要二次超参数化，且给出了基于神经切络核的分析，有助于综合比较不同缩放方案和初始化的重要性的差异。我们相信我们的研究结果对于更好地理解现代Transformer，特别是训练动态方面具有重要意义。

关于编码器型浅层变换器的收敛性研究