Despite the widespread adoption of Large language models (LLMs), their
remarkable capabilities remain limited to a few high-resource languages.
Additionally, many low-resource languages (e.g. African languages) are often
evaluated only on basic text classification tasks due to the lack of
appropriate or comprehensive benchmarks outside of high-resource languages. In
this paper, we introduce IrokoBench -- a human-translated benchmark dataset for
16 typologically-diverse low-resource African languages covering three tasks:
natural language inference~(AfriXNLI), mathematical reasoning~(AfriMGSM), and
multi-choice knowledge-based QA~(AfriMMLU). We use IrokoBench to evaluate
zero-shot, few-shot, and translate-test settings~(where test sets are
translated into English) across 10 open and four proprietary LLMs. Our
evaluation reveals a significant performance gap between high-resource
languages~(such as English and French) and low-resource African languages. We
observe a significant performance gap between open and proprietary models, with
the highest performing open model, Aya-101 only at 58\% of the best-performing
proprietary model GPT-4o performance. Machine translating the test set to
English before evaluation helped to close the gap for larger models that are
English-centric, like LLaMa 3 70B. These findings suggest that more efforts are
needed to develop and adapt LLMs for African languages.

本研究使用 IrokoBench 数据集对低资源的非洲语言进行了零样本学习、少样本学习和翻译测试，并发现了高资源语言和低资源非洲语言之间的显著性能差距。机器翻译测试集从而改善了一些以英语为中心的大型模型的表现，结果表明需要更多的努力来开发和适应非洲语言的大型语言模型。