Transformer models have advanced the state of the art in many Natural
Language Processing (NLP) tasks. In this paper, we present a new Transformer
architecture, Extended Transformer Construction (ETC), that addresses two key
challenges of standard Transformer architectures, namely scaling input length
and encoding structured inputs. To scale attention to longer inputs, we
introduce a novel global-local attention mechanism between global tokens and
regular input tokens. We also show that combining global-local attention with
relative position encodings and a Contrastive Predictive Coding (CPC)
pre-training objective allows ETC to encode structured inputs. We achieve
state-of-the-art results on four natural language datasets requiring long
and/or structured inputs.

本文介绍了一种新的 Transformer 架构 ——Extended Transformer Construction（ETC），该架构解决了标准 Transformer 架构的两个关键挑战，即扩展输入长度和编码结构化输入，并且通过将全局 - 局部注意力机制与相对位置编码以及对比预测编码（CPC）预训练目标结合使用，实现了对结构化输入的编码和处理，在需要长或结构化输入的四个自然语言数据集上取得了领先的成果。