Transformers have shown dominant performance across a range of domains
including language and vision. However, their computational cost grows
quadratically with the sequence length, making their usage prohibitive for
resource-constrained applications. To counter this, our approach is to divide
the whole sequence into segments. The information across segments can then be
aggregated using neurons with recurrence leveraging their inherent memory. Such
an approach leads to models with sequential processing capability at a lower
computation/memory cost. To investigate this idea, first, we examine the
effects of using local attention mechanism on the individual segments. Then we
propose a segmented recurrent transformer (SRformer) that combines segmented
attention with recurrent attention. It uses recurrent accumulate and fire (RAF)
layers to process information between consecutive segments. The loss caused by
reducing the attention window length is compensated by updating the product of
keys and values with RAF neurons' inherent recurrence. The segmented attention
and lightweight RAF gates ensure the efficiency of the proposed transformer. We
apply the proposed method to T5 and BART transformers. The modified models are
tested on summarization datasets including CNN-dailymail and XSUM. Notably,
using segmented inputs of different sizes, the proposed model achieves 4-19%
higher ROUGE1 scores than the segmented transformer baseline. Compared to full
attention, the proposed model largely reduces the complexity of cross attention
and results in around 40% reduction in computation cost.

本研究提出一种名为 “SRformer” 的分段循环变压器模型，通过使用分段关注机制和循环关注机制结合来处理长度较长的序列，降低计算和内存成本，并应用于 T5 和 BART 转换器上进行摘要生成等任务，结果显示比分段变压器基线具有更高的性能表现。