Recent advances in training multilingual language models on large datasets
seem to have shown promising results in knowledge transfer across languages and
achieve high performance on downstream tasks. However, we question to what
extent the current evaluation benchmarks and setups accurately measure
zero-shot cross-lingual knowledge transfer. In this work, we challenge the
assumption that high zero-shot performance on target tasks reflects high
cross-lingual ability by introducing more challenging setups involving
instances with multiple languages. Through extensive experiments and analysis,
we show that the observed high performance of multilingual models can be
largely attributed to factors not requiring the transfer of actual linguistic
knowledge, such as task- and surface-level knowledge. More specifically, we
observe what has been transferred across languages is mostly data artifacts and
biases, especially for low-resource languages. Our findings highlight the
overlooked drawbacks of existing cross-lingual test data and evaluation setups,
calling for a more nuanced understanding of the cross-lingual capabilities of
multilingual models.

通过实验证明多语言模型具有零 - shot 跨语言知识转移的能力并且在下游任务中表现出较高性能，但我们对当前评估基准和设置是否能准确衡量零 - shot 跨语言知识转移产生了质疑。本研究通过引入更具挑战性的多语言实例设置，表明多语言模型的高性能在很大程度上归因于不需要传递实际语言知识的因素，如任务和表面层知识。我们观察到跨语言传递的主要是数据工件和偏见，尤其是对于资源有限的语言。我们的发现凸显了现有跨语言测试数据和评估设置的缺点，呼吁对多语言模型的跨语言能力有更细致的理解。

多语言语言模型中跨语言知识迁移评估的分析

Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in  Multilingual Language Models

Researchers recently found out that sometimes language models achieve high
accuracy on benchmark data set, but they can not generalize very well with even
little changes to the original data set. This is sometimes due to data
artifacts, model is learning the spurious correlation between tokens and
labels, instead of the semantics and logic. In this work, we analyzed SNLI data
and visualized such spurious correlations. We proposed an adaptive up-sampling
algorithm to correct the data artifacts, which is simple and effective, and
does not need human edits or annotation. We did an experiment applying the
algorithm to fix the data artifacts in SNLI data and the model trained with
corrected data performed significantly better than the model trained with raw
SNLI data, overall, as well as on the subset we corrected.

通过分析 SNLI 数据集中的虚假关联并提出一种自适应上采样算法，在不需要人工编辑或注释的情况下纠正数据生成的错误，从而显著提高了模型在 SNLI 数据集上的性能。

减少数据伪影的自适应上采样算法

No More Distractions: an Adaptive Up-Sampling Algorithm to Reduce Data  Artifacts

Data artifacts incentivize machine learning models to learn non-transferable
generalizations by taking advantage of shortcuts in the data, and there is
growing evidence that data artifacts play a role for the strong results that
deep learning models achieve in recent natural language processing benchmarks.
In this paper, we focus on task-oriented dialogue and investigate whether
popular datasets such as MultiWOZ contain such data artifacts. We found that by
only keeping frequent phrases in the training examples, state-of-the-art models
perform similarly compared to the variant trained with full data, suggesting
they exploit these spurious correlations to solve the task. Motivated by this,
we propose a contrastive learning based framework to encourage the model to
ignore these cues and focus on learning generalisable patterns. We also
experiment with adversarial filtering to remove "easy" training instances so
that the model would focus on learning from the "harder" instances. We conduct
a number of generalization experiments -- e.g., cross-domain/dataset and
adversarial tests -- to assess the robustness of our approach and found that it
works exceptionally well.

本研究探讨了数据人工制品对机器学习模型泛化能力所产生的影响，提出一种基于对比学习的框架，试图鼓励深度学习模型学习普适规律并忽略数据人工制品。研究表明该方法具有很好的实验效果。