Zero-shot cross-lingual transfer learning has been shown to be highly
challenging for tasks involving a lot of linguistic specificities or when a
cultural gap is present between languages, such as in hate speech detection. In
this paper, we highlight this limitation for hate speech detection in several
domains and languages using strict experimental settings. Then, we propose to
train on multilingual auxiliary tasks -- sentiment analysis, named entity
recognition, and tasks relying on syntactic information -- to improve zero-shot
transfer of hate speech detection models across languages. We show how hate
speech detection models benefit from a cross-lingual knowledge proxy brought by
auxiliary tasks fine-tuning and highlight these tasks' positive impact on
bridging the hate speech linguistic and cultural gap between languages.

本文研究了零 - shot 语言跨域转移学习在对抗言检测等任务中所面临的挑战，并提出使用多语种辅助任务（情感分析、命名实体识别和依靠语法信息的任务）进行训练，以提高模型的零 - shot 转移到不同语言的能力，最终实现跨语言之间的言语和文化差异的缩小。

多语言辅助任务训练：缩小语言之间的差距，实现仇恨言论检测模型的零 - shot 迁移

Multilingual Auxiliary Tasks Training: Bridging the Gap between Languages for Zero-Shot Transfer of Hate Speech Detection Models

Pre-trained cross-lingual encoders such as mBERT (Devlin et al., 2019) and
XLMR (Conneau et al., 2020) have proven to be impressively effective at
enabling transfer-learning of NLP systems from high-resource languages to
low-resource languages. This success comes despite the fact that there is no
explicit objective to align the contextual embeddings of words/sentences with
similar meanings across languages together in the same space. In this paper, we
present a new method for learning multilingual encoders, AMBER (Aligned
Multilingual Bidirectional EncodeR). AMBER is trained on additional parallel
data using two explicit alignment objectives that align the multilingual
representations at different granularities. We conduct experiments on zero-shot
cross-lingual transfer learning for different tasks including sequence tagging,
sentence retrieval and sentence classification. Experimental results show that
AMBER obtains gains of up to 1.1 average F1 score on sequence tagging and up to
27.3 average accuracy on retrieval over the XLMR-large model which has 3.2x the
parameters of AMBER. Our code and models are available at
this http URL

本文提出了一种新的方法，即 AMBER（Aligned Multilingual Bidirectional EncodeR），它是一个学习对齐多语言编码器的方法，能够让预训练的跨语言编码器在不同的任务中获得更好的性能，如零 - shot 跨语言迁移学习和序列标注。

多语言双向编码器的显式对齐目标

Explicit Alignment Objectives for Multilingual Bidirectional Encoders

Multilingual contextual embeddings have demonstrated state-of-the-art
performance in zero-shot cross-lingual transfer learning, where multilingual
BERT is fine-tuned on one source language and evaluated on a different target
language. However, published results for mBERT zero-shot accuracy vary as much
as 17 points on the MLDoc classification task across four papers. We show that
the standard practice of using English dev accuracy for model selection in the
zero-shot setting makes it difficult to obtain reproducible results on the
MLDoc and XNLI tasks. English dev accuracy is often uncorrelated (or even
anti-correlated) with target language accuracy, and zero-shot performance
varies greatly at different points in the same fine-tuning run and between
different fine-tuning runs. These reproducibility issues are also present for
other tasks with different pre-trained embeddings (e.g., MLQA with XLM-R). We
recommend providing oracle scores alongside zero-shot results: still fine-tune
using English data, but choose a checkpoint with the target dev set. Reporting
this upper bound makes results more consistent by avoiding arbitrarily bad
checkpoints.

本文研究了多语言语境嵌入在零样本跨语言迁移学习上的表现，并发现使用英语进行零样本设置时很难获得可重复的结果和一致的检查点，建议在零样本结果旁边提供 oracle 分数，通过避免任意差的检查点使结果更一致。