The transformer model is known to be computationally demanding, and prohibitively costly for long sequences, as the self-attention module uses a quadratic time and space complexity with respect to sequence length. Many researchers have focused on designing new forms of self-attention or introducing new parameters to overcome this limitation, however a large portion of them prohibits the model to inherit weights from large pretrained models. In this work, the transformer's inefficiency has been taken care of from another perspective. We propose Fourier Transformer, a simple yet effective approach by progressively removing redundancies in hidden sequence using the ready-made Fast Fourier Transform (FFT) operator to perform Discrete Cosine Transformation (DCT). Fourier Transformer is able to significantly reduce computational costs while retain the ability to inherit from various large pretrained models. Experiments show that our model achieves state-of-the-art performances among all transformer-based models on the long-range modeling benchmark LRA with significant improvement in both speed and space. For generative seq-to-seq tasks including CNN/DailyMail and ELI5, by inheriting the BART weights our model outperforms the standard BART and other efficient models. \footnote{Our code is publicly available at \url{https://github.com/LUMIA-Group/FourierTransformer}}

本文提出了 Fourier Transformer，该方法通过使用现成的快速傅里叶变换（FFT）算子来执行离散余弦变换（DCT）来逐步消除隐藏序列中的冗余信息以大大减少计算成本，同时保留了从各种大型预训练模型中继承权重的能力。结果表明，该模型在长距离建模基准LRA上实现了所有基于Transformer的模型中的最佳性能，在速度和空间方面均有显着改进。对于包括CNN/DailyMail和ELI5在内的生成 Seq-to-Seq 任务，通过继承 BART 权重，我们的模型优于标准 BART 和其他高效模型。

傅里叶变换器：通过FFT算子消除序列冗余进行快速长距建模