Recent investigations into the inner-workings of state-of-the-art large-scale
pre-trained Transformer-based Natural Language Understanding (NLU) models
indicate that they appear to know humanlike syntax, at least to some extent. We
provide novel evidence that complicates this claim: we find that
state-of-the-art Natural Language Inference (NLI) models assign the same labels
to permuted examples as they do to the original, i.e. they are largely
invariant to random word-order permutations. This behavior notably differs from
that of humans; we struggle with ungrammatical sentences. To measure the
severity of this issue, we propose a suite of metrics and investigate which
properties of particular permutations lead models to be word-order invariant.
In the MNLI dataset, for example, we find almost all (98.7%) examples contain
at least one permutation which elicits the gold label. Models are sometimes
even able to assign gold labels to permutations that they originally failed to
predict correctly. We provide a comprehensive empirical evaluation of this
phenomenon, and further show that this issue exists for both Transformers and
pre-Transformer RNN / ConvNet based encoders, as well as across multiple
languages (English and Mandarin Chinese). Our code and data are available at
this https URL

通过一个明确的指标，我们提出了多种度量方法，并研究了哪些特定的排列方式使得大型 Transformer NLU 模型在自然语言推理任务中对随机词序具有一定程度的规律性。我们对此现象进行了综合实证评估，发现无论是基于 Transformer 还是基于 pre-Transformer RNN / ConvNet 结构的编码器，以及跨多种语言（英语和中文）都存在此问题，并提供相关数据和代码。