We show that Transformer encoder architectures can be sped up, with limited
accuracy costs, by replacing the self-attention sublayers with simple linear
transformations that "mix" input tokens. These linear mixers, along with
standard nonlinearities in feed-forward layers, prove competent at modeling
semantic relationships in several text classification tasks. Most surprisingly,
we find that replacing the self-attention sublayer in a Transformer encoder
with a standard, unparameterized Fourier Transform achieves 92-97% of the
accuracy of BERT counterparts on the GLUE benchmark, but trains 80% faster on
GPUs and 70% faster on TPUs at standard 512 input lengths. At longer input
lengths, our FNet model is significantly faster: when compared to the
"efficient" Transformers on the Long Range Arena benchmark, FNet matches the
accuracy of the most accurate models, while outpacing the fastest models across
all sequence lengths on GPUs (and across relatively shorter lengths on TPUs).
Finally, FNet has a light memory footprint and is particularly efficient at
smaller model sizes; for a fixed speed and accuracy budget, small FNet models
outperform Transformer counterparts.

通过使用线性混合器代替自注意层，与标准的非线性前馈层一起能够在几个文本分类任务中模拟语义关系。用标准、非参数特定的傅里叶变化替换 Transformer 编码器中的自注意子层在 GLUE 基准测试上取得了 92-97% 的 BERT 模型的准确性，同时在标准 512 输入长度的 GPU 上训练速度提高了 80%，在 TPU 上提高了 70%。在更长的输入长度上，FNet 模型速度极快，在长序列基准测试中，FNet 模型能够匹配最精确模型的准确性，提供更快的模型训练速度。此外，FNet 的内存占用相对较小，在较小的模型尺寸下运行的性能优于 Transformer 模型。