Understanding human language is one of the key themes of artificial
intelligence. For language representation, the capacity of effectively modeling
the linguistic knowledge from the detail-riddled and lengthy texts and getting
rid of the noises is essential to improve its performance. Traditional
attentive models attend to all words without explicit constraint, which results
in inaccurate concentration on some dispensable words. In this work, we propose
using syntax to guide the text modeling by incorporating explicit syntactic
constraints into attention mechanisms for better linguistically motivated word
representations. In detail, for self-attention network (SAN) sponsored
Transformer-based encoder, we introduce syntactic dependency of interest (SDOI)
design into the SAN to form an SDOI-SAN with syntax-guided self-attention.
Syntax-guided network (SG-Net) is then composed of this extra SDOI-SAN and the
SAN from the original Transformer encoder through a dual contextual
architecture for better linguistics inspired representation. The proposed
SG-Net is applied to typical Transformer encoders. Extensive experiments on
popular benchmark tasks, including machine reading comprehension, natural
language inference, and neural machine translation show the effectiveness of
the proposed SG-Net design.

本研究提出了一种基于句法导向机制的注意力机制，设计了一个语法指导的 self-attention 网络，用于自然语言处理等任务中，取得了较好的性能。