Since its introduction, the transformers architecture has seen great adoption
in NLP applications, but it also has limitations. Although the self-attention
mechanism allows for generating very rich representations of the input text,
its effectiveness may be limited in specialized domains such as legal, where,
for example, language models often have to process very long texts. In this
paper, we explore alternatives to replace the attention-based layers with
simpler token-mixing mechanisms: Hartley and Fourier transforms. Using these
non-parametric techniques, we train models with long input documents from
scratch in the legal domain setting. We also introduce a new hybrid Seq2Seq
architecture, a no-attention-based encoder connected with an attention-based
decoder, which performs quite well on existing summarization tasks with much
less compute and memory requirements. We believe that similar, if not better
performance, as in the case of long correlations of abstractive text
summarization tasks, can be achieved by adopting these simpler infrastructures.
This not only makes training models from scratch accessible to more people, but
also contributes to the reduction of the carbon footprint during training.

在本文中，我们探讨了用简单的记号混合机制 ——Hartley 和 Fourier 变换替代基于注意力的层的替代方法，并在法律领域设置中从头开始训练模型。我们还介绍了一种新的混合 Seq2Seq 架构，即无注意力的编码器与有注意力的解码器相连，该架构在现有的文摘任务中表现出色，并且计算和内存需求更低。我们相信，采用这些更简单的基础设施可以实现与抽象文本摘要任务中的长相关性类似甚至更好的性能。这不仅使更多人能够从头训练模型，而且还有助于减少训练过程中的碳足迹。