Although dominant in natural language processing, transformer-based models
remain challenged by the task of long-sequence processing, because the
computational cost of self-attention operations in transformers swells
quadratically with the input sequence length. To alleviate the complexity of
long-sequence processing, we propose a simple framework to enable the
offthe-shelf pre-trained transformers to process much longer sequences, while
the computation and memory costs remain growing linearly with the input
sequence lengths. More specifically, our method divides each long-sequence
input into a batch of chunks, then aligns the interchunk information during the
encoding steps, and finally selects the most representative hidden states from
the encoder for the decoding process. To extract inter-chunk semantic
information, we align the start and end token embeddings among chunks in each
encoding transformer block. To learn an effective hidden selection policy, we
design a dual updating scheme inspired by reinforcement learning, which regards
the decoders of transformers as environments, and the downstream performance
metrics as the rewards to evaluate the hidden selection actions. Our empirical
results on real-world long-text summarization and reading comprehension tasks
demonstrate effective improvements compared to prior longsequence processing
baselines.

通过将每个长序列输入分割成一批块，并在编码步骤中对块间信息进行对齐，我们提出了一个简单的框架，使通用的预训练 Transformer 能够处理更长的序列，同时计算和内存成本与输入序列长度线性增长。通过在每个编码 Transformer 模块中对块中的起始和结束标记嵌入进行对齐，我们能够提取块间语义信息。为了学习一个有效的隐藏状态选择策略，我们设计了一个受强化学习启发的双重更新方案，将 Transformer 的解码器视为环境，并将下游性能指标作为奖励来评估隐藏状态选择动作。我们在真实世界的长文本摘要和阅读理解任务上的实证结果表明，与之前的长序列处理基准相比，我们取得了有效的改进。