While significant progress has been made in benchmarking Large Language
Models (LLMs) across various tasks, there is a lack of comprehensive evaluation
of their abilities in responding to multi-turn instructions in less-commonly
tested languages like Arabic. Our paper offers a detailed examination of the
proficiency of open LLMs in such scenarios in Arabic. Utilizing a customized
Arabic translation of the MT-Bench benchmark suite, we employ GPT-4 as a
uniform evaluator for both English and Arabic queries to assess and compare the
performance of the LLMs on various open-ended tasks. Our findings reveal
variations in model responses on different task categories, e.g., logic vs.
literacy, when instructed in English or Arabic. We find that fine-tuned base
models using multilingual and multi-turn datasets could be competitive to
models trained from scratch on multilingual data. Finally, we hypothesize that
an ensemble of small, open LLMs could perform competitively to proprietary LLMs
on the benchmark.

在小众语言如阿拉伯语中，我们详细研究了开放式大型语言模型在回应多轮指令方面的能力，并利用自定义的阿拉伯文翻译的 MT-Bench 基准套件和 GPT-4 作为评估工具，对英文和阿拉伯文进行了综合评估和比较，结果表明在不同任务类别（逻辑和文学）以英文或阿拉伯文指令时，模型的回答会有所差异。我们发现，使用多语言和多轮次数据集进行微调的基础模型与从头开始使用多语言数据训练的模型具有竞争力。最后，我们假设一组小型、开放式大型语言模型在基准测试中具有与专有模型相当的性能。