The widespread absence of diacritical marks in Arabic text poses a
significant challenge for Arabic natural language processing (NLP). This paper
explores instances of naturally occurring diacritics, referred to as
"diacritics in the wild," to unveil patterns and latent information across six
diverse genres: news articles, novels, children's books, poetry, political
documents, and ChatGPT outputs. We present a new annotated dataset that maps
real-world partially diacritized words to their maximal full diacritization in
context. Additionally, we propose extensions to the analyze-and-disambiguate
approach in Arabic NLP to leverage these diacritics, resulting in notable
improvements. Our contributions encompass a thorough analysis, valuable
datasets, and an extended diacritization algorithm. We release our code and
datasets as open source.

通过探索阿拉伯语自然语言处理中的 “野生变音符号” 实例，本文提出了一个新的标注数据集，并提出了对阿拉伯语自然语言处理中的分析和消岐方法的扩展，以利用这些变音符号，从而实现显著的改进。

野外阿拉伯语音调符号：利用机会改进音调标记

Arabic Diacritics in the Wild: Exploiting Opportunities for Improved  Diacritization

In this paper, we address the significant gap in Arabic natural language
processing (NLP) resources by introducing ArabicaQA, the first large-scale
dataset for machine reading comprehension and open-domain question answering in
Arabic. This comprehensive dataset, consisting of 89,095 answerable and 3,701
unanswerable questions created by crowdworkers to look similar to answerable
ones, along with additional labels of open-domain questions marks a crucial
advancement in Arabic NLP resources. We also present AraDPR, the first dense
passage retrieval model trained on the Arabic Wikipedia corpus, specifically
designed to tackle the unique challenges of Arabic text retrieval. Furthermore,
our study includes extensive benchmarking of large language models (LLMs) for
Arabic question answering, critically evaluating their performance in the
Arabic language context. In conclusion, ArabicaQA, AraDPR, and the benchmarking
of LLMs in Arabic question answering offer significant advancements in the
field of Arabic NLP. The dataset and code are publicly accessible for further
research this https URL

本文介绍了 ArabicaQA，这是第一个用于阿拉伯语机器阅读理解和开放域问题回答的大规模数据集，包含了 89095 个可回答和 3701 个无法回答的问题，以及其他标记的开放域问题。同时还介绍了 AraDPR，这是第一个针对阿拉伯文本检索的密集段落检索模型，经过阿拉伯维基百科语料库的训练，专门用于解决阿拉伯文本检索中的独特挑战。此外，我们还对大型语言模型在阿拉伯语问题回答中进行了广泛的基准测试，并对其在阿拉伯语环境中的性能进行了批判性评估。总之，ArabicaQA、AraDPR 和在阿拉伯语问题回答中的语言模型基准测试为阿拉伯语自然语言处理领域带来了重要的进展。数据集和代码对于进一步研究都是公开可访问的。

ArabicaQA: 阿拉伯问答系统的综合数据集

ArabicaQA: A Comprehensive Dataset for Arabic Question Answering

The Arabic language is a morphologically rich language with relatively few
resources and a less explored syntax compared to English. Given these
limitations, Arabic Natural Language Processing (NLP) tasks like Sentiment
Analysis (SA), Named Entity Recognition (NER), and Question Answering (QA),
have proven to be very challenging to tackle. Recently, with the surge of
transformers based models, language-specific BERT based models have proven to
be very efficient at language understanding, provided they are pre-trained on a
very large corpus. Such models were able to set new standards and achieve
state-of-the-art results for most NLP tasks. In this paper, we pre-trained BERT
specifically for the Arabic language in the pursuit of achieving the same
success that BERT did for the English language. The performance of AraBERT is
compared to multilingual BERT from Google and other state-of-the-art
approaches. The results showed that the newly developed AraBERT achieved
state-of-the-art performance on most tested Arabic NLP tasks. The pretrained
araBERT models are publicly available on this https URL
hoping to encourage research and applications for Arabic NLP.

使用语言特定的 BERT 模型预训练，构建了 AraBERT 模型，以在阿拉伯语 NLP 任务方面实现最先进的表现。