Despite their successes in NLP, Transformer-based language models still
require extensive computing resources and suffer in low-resource or low-compute
settings. In this paper, we present AxomiyaBERTa, a novel BERT model for
Assamese, a morphologically-rich low-resource language (LRL) of Eastern India.
AxomiyaBERTa is trained only on the masked language modeling (MLM) task,
without the typical additional next sentence prediction (NSP) objective, and
our results show that in resource-scarce settings for very low-resource
languages like Assamese, MLM alone can be successfully leveraged for a range of
tasks. AxomiyaBERTa achieves SOTA on token-level tasks like Named Entity
Recognition and also performs well on "longer-context" tasks like Cloze-style
QA and Wiki Title Prediction, with the assistance of a novel embedding
disperser and phonological signals respectively. Moreover, we show that
AxomiyaBERTa can leverage phonological signals for even more challenging tasks,
such as a novel cross-document coreference task on a translated version of the
ECB+ corpus, where we present a new SOTA result for an LRL. Our source code and
evaluation scripts may be found at this https URL

本文提出了一种针对印度东部的形态丰富的低资源语言 (Assamese) 的 BERT 模型，名为 AxomiyaBERTa，该模型只在掩码语言模型 (MLM) 任务上进行训练，不包含典型的下一句预测 (NSP) 目标，结果显示在类似 Assamese 这样的低资源语言的资源稀缺环境中，仅使用 MLM 就可以成功地利用一系列任务，同时 AxomiyaBERTa 在诸如命名实体识别等基于 token 的任务上实现了 SOTA，在 Cloze-style QA 和 Wiki Title Prediction 等基于更长上下文的任务上也表现良好，其背后包含一个新型的嵌入扩散器和音韵信号。此外，本文还表明 AxomiyaBERTa 可以利用音韵信号处理更具挑战性的任务，例如在 ECB+ 语料库的翻译版本中进行新的交叉文件共指任务，我们在 LRL 上呈现了新的 SOTA 结果。