Incorporating hierarchical structures like constituency trees has been shown
to be effective for various natural language processing (NLP) tasks. However,
it is evident that state-of-the-art (SOTA) sequence-based models like the
Transformer struggle to encode such structures inherently. On the other hand,
dedicated models like the Tree-LSTM, while explicitly modeling hierarchical
structures, do not perform as efficiently as the Transformer. In this paper, we
attempt to bridge this gap with "Hierarchical Accumulation" to encode parse
tree structures into self-attention at constant time complexity. Our approach
outperforms SOTA methods in four IWSLT translation tasks and the WMT'14
English-German translation task. It also yields improvements over Transformer
and Tree-LSTM on three text classification tasks. We further demonstrate that
using hierarchical priors can compensate for data shortage, and that our model
prefers phrase-level attentions over token-level attentions.

本文提出了 “Hierarchy Accumulation” 的方法，将分析树结构编码为自注意力，以常数时间复杂度实现了序列模型，相较于 SOTA 方法，在四个 IWSLT 翻译任务和 WMT'14 英德翻译任务上表现更好，并在三项文本分类任务上胜过 Transformer 和 Tree-LSTM，同时表明使用分层先验可以补偿数据短缺，该模型更倾向于使用短语级别的自注意力。