Pre-trained Transformer models have achieved successes in a wide range of NLP
tasks, but are inefficient when dealing with long input sequences. Existing
studies try to overcome this challenge via segmenting the long sequence
followed by hierarchical encoding or post-hoc aggregation. We propose a
synchronization mechanism for hierarchical encoding. Our approach first
identifies anchor tokens across segments and groups them by their roles in the
original input sequence. Then inside Transformer layer, anchor embeddings are
synchronized within their group via a self-attention module. Our approach is a
general framework with sufficient flexibility -- when adapted to a new task, it
is easy to be enhanced with the task-specific anchor definitions. Experiments
on two representative tasks with different types of long input texts,
NarrativeQA summary setting and wild multi-hop reasoning from HotpotQA,
demonstrate that our approach is able to improve the global information
exchange among segments while maintaining efficiency.

本研究提出一种基于同步机制的层次编码方法，将长的输入序列分段处理并在 Transformer 层内利用自注意力机制对锚点嵌入进行同步，从而改善了不同类型长输入文本数据的全局信息交流。