Despite that Transformers perform well in NLP tasks, recent studies suggest
that self-attention is theoretically limited in learning even some regular and
context-free languages. These findings motivated us to think about their
implications in modeling natural language, which is hypothesized to be mildly
context-sensitive. We test Transformer's ability to learn a variety of mildly
context-sensitive languages of varying complexities, and find that they
generalize well to unseen in-distribution data, but their ability to
extrapolate to longer strings is worse than that of LSTMs. Our analyses show
that the learned self-attention patterns and representations modeled dependency
relations and demonstrated counting behavior, which may have helped the models
solve the languages.

尽管 Transformer 在自然语言处理任务中表现良好，但最近的研究表明自我注意力在学习一些常规和无上下文语言时在理论上存在限制。我们测试了 Transformer 学习各种复杂性的轻度上下文敏感语言的能力，发现它们对未见过的分布数据具有良好的泛化能力，但它们对较长字符串的外推能力不及 LSTMs。我们的分析表明，学得的自我注意力模式和表示模拟了依赖关系并展示了计数行为，这可能有助于模型解决这些语言。