Recent studies of the emergent capabilities of transformer-based Natural
Language Understanding (NLU) models have indicated that they have an
understanding of lexical and compositional semantics. We provide evidence that
suggests these claims should be taken with a grain of salt: we find that
state-of-the-art Natural Language Inference (NLI) models are sensitive towards
minor semantics preserving surface-form variations, which lead to sizable
inconsistent model decisions during inference. Notably, this behaviour differs
from valid and in-depth comprehension of compositional semantics, however does
neither emerge when evaluating model accuracy on standard benchmarks nor when
probing for syntactic, monotonic, and logically robust reasoning. We propose a
novel framework to measure the extent of semantic sensitivity. To this end, we
evaluate NLI models on adversarially generated examples containing minor
semantics-preserving surface-form input noise. This is achieved using
conditional text generation, with the explicit condition that the NLI model
predicts the relationship between the original and adversarial inputs as a
symmetric equivalence entailment. We systematically study the effects of the
phenomenon across NLI models for \emph{in-} and \emph{out-of} domain settings.
Our experiments show that semantic sensitivity causes performance degradations
of $12.92\%$ and $23.71\%$ average over \emph{in-} and \emph{out-of-} domain
settings, respectively. We further perform ablation studies, analysing this
phenomenon across models, datasets, and variations in inference and show that
semantic sensitivity can lead to major inconsistency within model predictions.

通过评估自然语言推理模型对含有微小语义保留表面形式噪声的对抗生成样本的效果，我们提供了证据表明，最新基于转换器的自然语言理解模型对于细微的语义保留有敏感性，这导致了推理过程中明显的不一致性。这种语义敏感性会导致在模型预测中的性能降低 12.92% 和 23.71%。

语义敏感度与不一致预测：衡量 NLI 模型的脆弱性

Semantic Sensitivities and Inconsistent Predictions: Measuring the  Fragility of NLI Models

Most research on hate speech detection has focused on English where a
sizeable amount of labeled training data is available. However, to expand hate
speech detection into more languages, approaches that require minimal training
data are needed. In this paper, we test whether natural language inference
(NLI) models which perform well in zero- and few-shot settings can benefit hate
speech detection performance in scenarios where only a limited amount of
labeled data is available in the target language. Our evaluation on five
languages demonstrates large performance improvements of NLI fine-tuning over
direct fine-tuning in the target language. However, the effectiveness of
previous work that proposed intermediate fine-tuning on English data is hard to
match. Only in settings where the English training data does not match the test
domain, can our customised NLI-formulation outperform intermediate fine-tuning
on English. Based on our extensive experiments, we propose a set of
recommendations for hate speech detection in languages where minimal labeled
training data is available.

本研究探讨了自然语言推理（NLI）模型对于在仅有少量标记数据的情况下，提高仇恨言论检测性能的可行性，并提出了一组针对存在极少标记训练数据的语言的仇恨言论检测推荐方法。