We propose a novel framework for cross-lingual content flagging with limited
target-language data, which significantly outperforms prior work in terms of
predictive performance. The framework is based on a nearest-neighbour
architecture. It is a modern instantiation of the vanilla k-nearest neighbour
model, as we use Transformer representations in all its components. Our
framework can adapt to new source-language instances, without the need to be
retrained from scratch. Unlike prior work on neighbourhood-based approaches, we
encode the neighbourhood information based on query--neighbour interactions. We
propose two encoding schemes and we show their effectiveness using both
qualitative and quantitative analysis. Our evaluation results on eight
languages from two different datasets for abusive language detection show
sizable improvements of up to 9.5 F1 points absolute (for Italian) over strong
baselines. On average, we achieve 3.6 absolute F1 points of improvement for the
three languages in the Jigsaw Multilingual dataset and 2.14 points for the WUL
dataset.

我们提出了一种使用有限的目标语言数据进行跨语言内容标记的新框架，采用最近邻居结构，在所有组件中使用 Transformer 表示法，可适应新的源语言实例，提出两种编码方案并在滥用语言检测方面实现了显著的改进。

面向资源匮乏情况下内容标记的邻域框架

A Neighbourhood Framework for Resource-Lean Content Flagging

We re-evaluate the standard practice of sharing weights between input and
output embeddings in state-of-the-art pre-trained language models. We show that
decoupled embeddings provide increased modeling flexibility, allowing us to
significantly improve the efficiency of parameter allocation in the input
embedding of multilingual models. By reallocating the input embedding
parameters in the Transformer layers, we achieve dramatically better
performance on standard natural language understanding tasks with the same
number of parameters during fine-tuning. We also show that allocating
additional capacity to the output embedding provides benefits to the model that
persist through the fine-tuning stage even though the output embedding is
discarded after pre-training. Our analysis shows that larger output embeddings
prevent the model's last layers from overspecializing to the pre-training task
and encourage Transformer representations to be more general and more
transferable to other tasks and languages. Harnessing these findings, we are
able to train models that achieve strong performance on the XTREME benchmark
without increasing the number of parameters at the fine-tuning stage.

重新评估了先进的预训练语言模型中共享输入和输出嵌入权重的标准做法，表明解耦的嵌入提供了更好的建模灵活性，允许我们在多语言模型的输入嵌入的参数分配方面显着提高参数分配效率。通过在 Transformer 层中重新分配输入嵌入参数，我们在保持微调期间相同的参数数量的情况下，实现了标准自然语言理解任务的极大性能改进。Allocate 额外的容量来提供模型的输出嵌入即使在预训练后被丢弃，也能在微调阶段带来好处。我们的分析表明，更大的输出嵌入可以防止模型的最后几层过度专注于预训练任务，并鼓励 Transformer 表示更加普遍和更易于传递到其他任务和语言。利用这些发现，我们能够训练出在 XTREME 基准上表现出色的模型，而不增加微调阶段的参数数量。