We introduce Transformer Grammars (TGs), a novel class of Transformer
language models that combine (i) the expressive power, scalability, and strong
performance of Transformers and (ii) recursive syntactic compositions, which
here are implemented through a special attention mask and deterministic
transformation of the linearized tree. We find that TGs outperform various
strong baselines on sentence-level language modeling perplexity, as well as on
multiple syntax-sensitive language modeling evaluation metrics. Additionally,
we find that the recursive syntactic composition bottleneck which represents
each sentence as a single vector harms perplexity on document-level language
modeling, providing evidence that a different kind of memory mechanism -- one
that is independent of composed syntactic representations -- plays an important
role in current successful models of long text.

Transformer Grammars 是一种新颖的语言模型，通过特殊的注意力掩码和确定性转换实现递归句法组合，提高了句子级别和句法敏感的语言建模性能，在长文本建模中，递归的句法组合对表示整个句子向量造成了瓶颈并影响了逼近度，表明一个独立于组合句法表示的不同类型的记忆机制在当前成功的模型中发挥了重要作用。