The ubiquity of offensive content on social media is a growing cause for
concern among companies and government organizations. Recently,
transformer-based models such as BERT, XLNET, and XLM-R have achieved
state-of-the-art performance in detecting various forms of offensive content
(e.g. hate speech, cyberbullying, and cyberaggression). However, the majority
of these models are limited in their capabilities due to their encoder-only
architecture, which restricts the number and types of labels in downstream
tasks. Addressing these limitations, this study presents the first pre-trained
model with encoder-decoder architecture for offensive language identification
with text-to-text transformers (T5) trained on two large offensive language
identification datasets; SOLID and CCTK. We investigate the effectiveness of
combining two datasets and selecting an optimal threshold in semi-supervised
instances in SOLID in the T5 retraining step. Our pre-trained T5 model
outperforms other transformer-based models fine-tuned for offensive language
detection, such as fBERT and HateBERT, in multiple English benchmarks.
Following a similar approach, we also train the first multilingual pre-trained
model for offensive language identification using mT5 and evaluate its
performance on a set of six different languages (German, Hindi, Korean,
Marathi, Sinhala, and Spanish). The results demonstrate that this multilingual
model achieves a new state-of-the-art on all the above datasets, showing its
usefulness in multilingual scenarios. Our proposed T5-based models will be made
freely available to the community.

社交媒体上冒犯内容的普遍存在是公司和政府组织越来越关注的问题。本研究提出了第一个具有编码器 - 解码器结构的预训练模型，用于冒犯语言识别，并在两个大型冒犯语言识别数据集（SOLID 和 CCTK）上训练。研究结果表明，预训练的 T5 模型在多个英文基准测试中优于其他基于变压器的模型，在多语言场景中，多语言预训练模型在所有上述数据集上实现了新的最优表现。

多语言冒犯性语言识别的文本对文本模型

A Text-to-Text Model for Multilingual Offensive Language Identification

The data scarcity in low-resource languages has become a bottleneck to
building robust neural machine translation systems. Fine-tuning a multilingual
pre-trained model (e.g., mBART (Liu et al., 2020)) on the translation task is a
good approach for low-resource languages; however, its performance will be
greatly limited when there are unseen languages in the translation pairs. In
this paper, we present a continual pre-training (CPT) framework on mBART to
effectively adapt it to unseen languages. We first construct noisy
mixed-language text from the monolingual corpus of the target language in the
translation pair to cover both the source and target languages, and then, we
continue pre-training mBART to reconstruct the original monolingual text.
Results show that our method can consistently improve the fine-tuning
performance upon the mBART baseline, as well as other strong baselines, across
all tested low-resource translation pairs containing unseen languages.
Furthermore, our approach also boosts the performance on translation pairs
where both languages are seen in the original mBART's pre-training. The code is
available at this https URL

本文提出了一种连续预训练 (CPT) 框架来应对低资源语言中看不见的语言，通过使用目标语言的单语语料库构建混合语言文本并继续对 mBART 进行预训练以优化低资源翻译任务，结果表明我们的方法能够持续提高翻译性能。