Recently, large language models (LLMs) fine-tuned to follow human instruction
have exhibited significant capabilities in various English NLP tasks. However,
their performance in grammatical error correction (GEC) tasks, particularly in
non-English languages, remains significantly unexplored. In this paper, we
delve into abilities of instruction fine-tuned LLMs in Arabic GEC, a task made
complex due to Arabic's rich morphology. Our findings suggest that various
prompting methods, coupled with (in-context) few-shot learning, demonstrate
considerable effectiveness, with GPT-4 achieving up to $65.49$
F\textsubscript{1} score under expert prompting (approximately $5$ points
higher than our established baseline). This highlights the potential of LLMs in
low-resource settings, offering a viable approach for generating useful
synthetic data for model training. Despite these positive results, we find that
instruction fine-tuned models, regardless of their size, significantly
underperform compared to fully fine-tuned models of significantly smaller
sizes. This disparity highlights a substantial room for improvements for LLMs.
Inspired by methods from low-resource machine translation, we also develop a
method exploiting synthetic data that significantly outperforms previous models
on two standard Arabic benchmarks. Our work sets new SoTA for Arabic GEC, with
$72.19\%$ and $73.26$ F$_{1}$ on the 2014 and 2015 QALB datasets, respectively.

近期，将大型语言模型 fine-tuned 后用于人类指令的研究表明在各种英语自然语言处理任务上取得了显著的能力，但在语法错误纠正任务中，特别是在非英语语言中的应用仍有待深入探索。本文研究了 fine-tuned 后的语言模型在阿拉伯语语法错误纠正中的能力，该任务由于阿拉伯语的丰富形态而变得复杂。我们的研究结果表明，各种提示方法与（上下文中的）少样本学习相结合，具有明显的效果，GPT-4 在专家提示下的 F	extsubscript {1} 分数可达到 65.49（比我们建立的基线高出约 5 个点），这凸显了语言模型在低资源环境中的潜力，为模型训练生成有用的合成数据提供了可行的方法。尽管取得了这些积极的结果，我们发现无论大小如何，fine-tuned 后的模型在性能上显著低于大小显著较小的完全 fine-tuned 模型，这种差异突显了语言模型改进的重要空间。受低资源机器翻译方法的启发，我们还开发了一种利用合成数据的方法，该方法在两个标准阿拉伯语基准测试上显著优于先前的模型。我们的工作在阿拉伯语语法错误纠正领域取得了新的最佳结果，分别达到了 2014 和 2015 年 QALB 数据集的 72.19% 和 73.26 的 F$_{1}$ 值。

ChatGPT 对阿拉伯语语法错误修正

ChatGPT for Arabic Grammatical Error Correction

As the breadth and depth of language model applications continue to expand
rapidly, it is increasingly important to build efficient frameworks for
measuring and mitigating the learned or inherited social biases of these
models. In this paper, we present our work on evaluating instruction fine-tuned
language models' ability to identify bias through zero-shot prompting,
including Chain-of-Thought (CoT) prompts. Across LLaMA and its two instruction
fine-tuned versions, Alpaca 7B performs best on the bias identification task
with an accuracy of 56.7%. We also demonstrate that scaling up LLM size and
data diversity could lead to further performance gain. This is a
work-in-progress presenting the first component of our bias mitigation
framework. We will keep updating this work as we get more results.

评估指令微调语言模型通过零样本提示，包括思维链（CoT）提示，识别偏见的能力，其中 Alpaca 7B 在偏见识别任务中具有 56.7％的准确性，同时展示了 LLM 规模和数据多样性的扩大可能会进一步提高性能。