The use of multilingual language models for tasks in low and high-resource
languages has been a success story in deep learning. In recent times, Arabic
has been receiving widespread attention on account of its dialectal variance.
While prior research studies have tried to adapt these multilingual models for
dialectal variants of Arabic, it still remains a challenging problem owing to
the lack of sufficient monolingual dialectal data and parallel translation data
of such dialectal variants. It remains an open problem on whether the limited
dialectical data can be used to improve the models trained in Arabic on its
dialectal variants. First, we show that multilingual-BERT (mBERT) incrementally
pretrained on Arabic monolingual data takes less training time and yields
comparable accuracy when compared to our custom monolingual Arabic model and
beat existing models (by an avg metric of +$6.41$). We then explore two
continual pre-training methods -- (1) using small amounts of dialectical data
for continual finetuning and (2) parallel Arabic to English data and a
Translation Language Modeling loss function. We show that both approaches help
improve performance on dialectal classification tasks ($+4.64$ avg. gain) when
used on monolingual models.

在使用多语言语言模型进行低资源和高资源语言的任务中，阿拉伯语的方言差异引起了广泛关注。本文采用 mBERT 预训练和两种持续预训练方法，证明这些方法都可以帮助提高方言分类任务的性能。

面向阿拉伯方言变异鲁棒性的参数和数据高效连续预训练

Parameter and Data Efficient Continual Pre-training for Robustness to Dialectal Variance in Arabic

Hope Speech Detection, a task of recognizing positive expressions, has made
significant strides recently. However, much of the current works focus on model
development without considering the issue of inherent imbalance in the data.
Our work revisits this issue in hope-speech detection by introducing focal
loss, data augmentation, and pre-processing strategies. Accordingly, we find
that introducing focal loss as part of Multilingual-BERT's (M-BERT) training
process mitigates the effect of class imbalance and improves overall F1-Macro
by 0.11. At the same time, contextual and back-translation-based word
augmentation with M-BERT improves results by 0.10 over baseline despite
imbalance. Finally, we show that overlapping word removal based on
pre-processing, though simple, improves F1-Macro by 0.28. In due process, we
present detailed studies depicting various behaviors of each of these
strategies and summarize key findings from our empirical results for those
interested in getting the most out of M-BERT for hope speech detection under
real-world conditions of data imbalance.

本研究利用聚焦损失、数据增强和预处理策略等方式，解决了积极表达检测中数据不平衡的问题，并通过详细研究发现，将聚焦损失应用于多语言 BERT 的训练过程中，弥补了类别不平衡的影响，提高了整体的 F1-Macro 值，而基于上下文和反向翻译的词语增强和基于预处理的去重复方法也分别使结果比基线模型提升了 0.10 和 0.28。本研究总结了每种方法的性能及实现过程，适用于在真实世界下解决希望言论检测中数据不平衡的问题。