Normalization layers are ubiquitous in modern neural networks and have long been considered essential. This work demonstrates that Transformers without normalization can achieve the same or better performance using a remarkably simple technique. We introduce Dynamic Tanh (DyT), an element-wise operation $DyT($x$) = \tanh(\alpha $x$)$, as a drop-in replacement for normalization layers in Transformers. DyT is inspired by the observation that layer normalization in Transformers often produces tanh-like, $S$-shaped input-output mappings. By incorporating DyT, Transformers without normalization can match or exceed the performance of their normalized counterparts, mostly without hyperparameter tuning. We validate the effectiveness of Transformers with DyT across diverse settings, ranging from recognition to generation, supervised to self-supervised learning, and computer vision to language models. These findings challenge the conventional understanding that normalization layers are indispensable in modern neural networks, and offer new insights into their role in deep networks.

本研究解决了现代神经网络中归一化层被普遍认为不可或缺的问题。我们提出了一种名为动态双曲正切（DyT）的新方法，作为变换器中归一化层的替代，显示出无归一化的变换器依然能够实现相同或更好的性能。最重要的发现是，DyT的引入使得变换器在多个学习设置中表现优异，挑战了归一化层在深度网络中的传统地位。