Transformers have achieved great success in many artificial intelligence
fields, such as natural language processing, computer vision, and audio
processing. Therefore, it is natural to attract lots of interest from academic
and industry researchers. Up to the present, a great variety of Transformer
variants (a.k.a. X-formers) have been proposed, however, a systematic and
comprehensive literature review on these Transformer variants is still missing.
In this survey, we provide a comprehensive review of various X-formers. We
first briefly introduce the vanilla Transformer and then propose a new taxonomy
of X-formers. Next, we introduce the various X-formers from three perspectives:
architectural modification, pre-training, and applications. Finally, we outline
some potential directions for future research.

本文综述了各种 X-formers，包括 Transformer 的变体，提出了一个新的分类法并从结构修改、预训练和应用三个方面介绍了各种 X-formers，并提出了未来研究的一些潜在方向。

Transformers 综述

A Survey of Transformers

Training Generative Adversarial Networks (GANs) is notoriously challenging.
We propose and study an architectural modification, self-modulation, which
improves GAN performance across different data sets, architectures, losses,
regularizers, and hyperparameter settings. Intuitively, self-modulation allows
the intermediate feature maps of a generator to change as a function of the
input noise vector. While reminiscent of other conditioning techniques, it
requires no labeled data. In a large-scale empirical study we observe a
relative decrease of $5\%-35\%$ in FID. Furthermore, all else being equal,
adding this modification to the generator leads to improved performance in
$124/144$ ($86\%$) of the studied settings. Self-modulation is a simple
architectural change that requires no additional parameter tuning, which
suggests that it can be applied readily to any GAN.

本文提出了一种简单的结构性改进 self-modulation，以提高生成对抗网络 (GAN) 的性能，并证明其可以适用于不同的数据集、架构、损失函数、正则化和超参数设置，大规模的实证研究表明，相对 FID 降低了 5%~35%，并且在 124/144 的研究设置中提高了 GAN 的性能