Recent studies of the emergent capabilities of transformer-based Natural
Language Understanding (NLU) models have indicated that they have an
understanding of lexical and compositional semantics. We provide evidence that
suggests these claims should be taken with a grain of salt: we find that
state-of-the-art Natural Language Inference (NLI) models are sensitive towards
minor semantics preserving surface-form variations, which lead to sizable
inconsistent model decisions during inference. Notably, this behaviour differs
from valid and in-depth comprehension of compositional semantics, however does
neither emerge when evaluating model accuracy on standard benchmarks nor when
probing for syntactic, monotonic, and logically robust reasoning. We propose a
novel framework to measure the extent of semantic sensitivity. To this end, we
evaluate NLI models on adversarially generated examples containing minor
semantics-preserving surface-form input noise. This is achieved using
conditional text generation, with the explicit condition that the NLI model
predicts the relationship between the original and adversarial inputs as a
symmetric equivalence entailment. We systematically study the effects of the
phenomenon across NLI models for \emph{in-} and \emph{out-of} domain settings.
Our experiments show that semantic sensitivity causes performance degradations
of $12.92\%$ and $23.71\%$ average over \emph{in-} and \emph{out-of-} domain
settings, respectively. We further perform ablation studies, analysing this
phenomenon across models, datasets, and variations in inference and show that
semantic sensitivity can lead to major inconsistency within model predictions.

通过评估自然语言推理模型对含有微小语义保留表面形式噪声的对抗生成样本的效果，我们提供了证据表明，最新基于转换器的自然语言理解模型对于细微的语义保留有敏感性，这导致了推理过程中明显的不一致性。这种语义敏感性会导致在模型预测中的性能降低 12.92% 和 23.71%。