Toxic comment detection on social media has proven to be essential for
content moderation. This paper compares a wide set of different models on a
highly skewed multi-label hate speech dataset. We consider inference time and
several metrics to measure performance and bias in our comparison. We show that
all BERTs have similar performance regardless of the size, optimizations or
language used to pre-train the models. RNNs are much faster at inference than
any of the BERT. BiLSTM remains a good compromise between performance and
inference time. RoBERTa with Focal Loss offers the best performance on biases
and AUROC. However, DistilBERT combines both good AUROC and a low inference
time. All models are affected by the bias of associating identities. BERT, RNN,
and XLNet are less sensitive than the CNN and Compact Convolutional
Transformers.

比较多个模型在高度倾斜的多标签仇恨言论数据集上进行毒性评论检测，结果表明 BERT、RNN 和 XLNet 对关联身份的偏见较不敏感，RoBERTa 的 Focal Loss 表现最佳，而 DistilBERT 则结合了良好的 AUROC 和低推理时间。

Civil Comments 数据集上有害评论分类的基准

A benchmark for toxic comment classification on Civil Comments dataset

Given the impact of language models on the field of Natural Language
Processing, a number of Spanish encoder-only masked language models (aka BERTs)
have been trained and released. These models were developed either within large
projects using very large private corpora or by means of smaller scale academic
efforts leveraging freely available data. In this paper we present a
comprehensive head-to-head comparison of language models for Spanish with the
following results: (i) Previously ignored multilingual models from large
companies fare better than monolingual models, substantially changing the
evaluation landscape of language models in Spanish; (ii) Results across the
monolingual models are not conclusive, with supposedly smaller and inferior
models performing competitively. Based on these empirical results, we argue for
the need of more research to understand the factors underlying them. In this
sense, the effect of corpus size, quality and pre-training techniques need to
be further investigated to be able to obtain Spanish monolingual models
significantly better than the multilingual ones released by large private
companies, specially in the face of rapid ongoing progress in the field. The
recent activity in the development of language technology for Spanish is to be
welcomed, but our results show that building language models remains an open,
resource-heavy problem which requires to marry resources (monetary and/or
computational) with the best research expertise and practice.

本文提供了一份全面的研究报告，比较了用于西班牙语的语言模型，报告显示：先前被忽略的大型跨语言模型比单语模型表现更好，单语模型的结果并不是确定性的，需要进一步研究语料库大小、质量和预训练技术等因素。

西班牙语语言模型评估中得到的经验教训

Lessons learned from the evaluation of Spanish Language Models

Large-scale pretraining and task-specific fine-tuning is now the standard
methodology for many tasks in computer vision and natural language processing.
Recently, a multitude of methods have been proposed for pretraining vision and
language BERTs to tackle challenges at the intersection of these two key areas
of AI. These models can be categorised into either single-stream or dual-stream
encoders. We study the differences between these two categories, and show how
they can be unified under a single theoretical framework. We then conduct
controlled experiments to discern the empirical differences between five V&L
BERTs. Our experiments show that training data and hyperparameters are
responsible for most of the differences between the reported results, but they
also reveal that the embedding layer plays a crucial role in these massive
models.

本文探讨了计算机视觉和自然语言处理中预训练方法的标准方法学习经验，分别采用单流和双流编码器进行归纳分析。我们就五种视觉与语言 BERT 的差异进行经验研究，揭示出预训练数据和超参数是这些异常壮观的模型之间不同的原因，同时将两类网络统一在同一理论框架之下，指出 embedding layer 在这些模型中具有至关重要的作用。