Relations between words are governed by hierarchical structure rather than
linear ordering. Sequence-to-sequence (seq2seq) models, despite their success
in downstream NLP applications, often fail to generalize in a
hierarchy-sensitive manner when performing syntactic transformations - for
example, transforming declarative sentences into questions. However, syntactic
evaluations of seq2seq models have only observed models that were not
pre-trained on natural language data before being trained to perform syntactic
transformations, in spite of the fact that pre-training has been found to
induce hierarchical linguistic generalizations in language models; in other
words, the syntactic capabilities of seq2seq models may have been greatly
understated. We address this gap using the pre-trained seq2seq models T5 and
BART, as well as their multilingual variants mT5 and mBART. We evaluate whether
they generalize hierarchically on two transformations in two languages:
question formation and passivization in English and German. We find that
pre-trained seq2seq models generalize hierarchically when performing syntactic
transformations, whereas models trained from scratch on syntactic
transformations do not. This result presents evidence for the learnability of
hierarchical syntactic information from non-annotated natural language text
while also demonstrating that seq2seq models are capable of syntactic
generalization, though only after exposure to much more language data than
human learners receive.

本文通过使用预训练的 seq2seq 模型，探究其是否能够在句法变换中进行层次化泛化，同时还证明了从未经注释的自然语言文本中可以学习到分层句法信息，_SEQ2SEQ 模型具有句法泛化能力，但需要比人类学习者接收更多的语言数据才能达到该能力_。