Large language models (LLMs) finetuned to follow human instruction have
recently exhibited significant capabilities in various English NLP tasks.
However, their performance in grammatical error correction (GEC), especially on
languages other than English, remains significantly unexplored. In this work,
we evaluate the abilities of instruction finetuned LLMs in Arabic GEC, a
complex task due to Arabic's rich morphology. Our findings suggest that various
prompting methods, coupled with (in-context) few-shot learning, demonstrate
considerable effectiveness, with GPT-4 achieving up to $65.49$ F$_{1}$ score
under expert prompting (approximately $5$ points higher than our established
baseline). Despite these positive results, we find that instruction finetuned
models, regardless of their size, are still outperformed by fully finetuned
ones, even if they are significantly smaller in size. This disparity highlights
substantial room for improvements for LLMs. Inspired by methods used in
low-resource machine translation, we also develop a method exploiting synthetic
data that significantly outperforms previous models on two standard Arabic
benchmarks. Our best model achieves a new SOTA on Arabic GEC, with $73.29$ and
$73.26$ F$_{1}$ on the 2014 and 2015 QALB datasets, respectively, compared to
peer-reviewed published baselines.

对于阿拉伯语的语法错误纠正任务，研究发现使用各种提示方法和少量样本学习的指令微调大型语言模型具有显著的效果，但是无论规模大小，指令微调模型的性能仍然不及完全微调的模型，此差距指出了语言模型改进的空间。同时，借鉴低资源机器翻译方法，利用合成数据也能显著提升阿拉伯语语法错误纠正的性能， 在两个标准阿拉伯语数据集上创造了新的最佳结果，分别达到 73.29 和 73.26 的 F1 分数，相比于同行评审发表的基准模型。

超越英语：评估用于阿拉伯文语法错误纠正的 LLMs

Beyond English: Evaluating LLMs for Arabic Grammatical Error Correction

Recently, large language models (LLMs) fine-tuned to follow human instruction
have exhibited significant capabilities in various English NLP tasks. However,
their performance in grammatical error correction (GEC) tasks, particularly in
non-English languages, remains significantly unexplored. In this paper, we
delve into abilities of instruction fine-tuned LLMs in Arabic GEC, a task made
complex due to Arabic's rich morphology. Our findings suggest that various
prompting methods, coupled with (in-context) few-shot learning, demonstrate
considerable effectiveness, with GPT-4 achieving up to $65.49$
F\textsubscript{1} score under expert prompting (approximately $5$ points
higher than our established baseline). This highlights the potential of LLMs in
low-resource settings, offering a viable approach for generating useful
synthetic data for model training. Despite these positive results, we find that
instruction fine-tuned models, regardless of their size, significantly
underperform compared to fully fine-tuned models of significantly smaller
sizes. This disparity highlights a substantial room for improvements for LLMs.
Inspired by methods from low-resource machine translation, we also develop a
method exploiting synthetic data that significantly outperforms previous models
on two standard Arabic benchmarks. Our work sets new SoTA for Arabic GEC, with
$72.19\%$ and $73.26$ F$_{1}$ on the 2014 and 2015 QALB datasets, respectively.

近期，将大型语言模型 fine-tuned 后用于人类指令的研究表明在各种英语自然语言处理任务上取得了显著的能力，但在语法错误纠正任务中，特别是在非英语语言中的应用仍有待深入探索。本文研究了 fine-tuned 后的语言模型在阿拉伯语语法错误纠正中的能力，该任务由于阿拉伯语的丰富形态而变得复杂。我们的研究结果表明，各种提示方法与（上下文中的）少样本学习相结合，具有明显的效果，GPT-4 在专家提示下的 F	extsubscript {1} 分数可达到 65.49（比我们建立的基线高出约 5 个点），这凸显了语言模型在低资源环境中的潜力，为模型训练生成有用的合成数据提供了可行的方法。尽管取得了这些积极的结果，我们发现无论大小如何，fine-tuned 后的模型在性能上显著低于大小显著较小的完全 fine-tuned 模型，这种差异突显了语言模型改进的重要空间。受低资源机器翻译方法的启发，我们还开发了一种利用合成数据的方法，该方法在两个标准阿拉伯语基准测试上显著优于先前的模型。我们的工作在阿拉伯语语法错误纠正领域取得了新的最佳结果，分别达到了 2014 和 2015 年 QALB 数据集的 72.19% 和 73.26 的 F$_{1}$ 值。