As large language models (LLMs) see increasing adoption across the globe, it
is imperative for LLMs to be representative of the linguistic diversity of the
world. India is a linguistically diverse country of 1.4 Billion people. To
facilitate research on multilingual LLM evaluation, we release IndicGenBench -
the largest benchmark for evaluating LLMs on user-facing generation tasks
across a diverse set 29 of Indic languages covering 13 scripts and 4 language
families. IndicGenBench is composed of diverse generation tasks like
cross-lingual summarization, machine translation, and cross-lingual question
answering. IndicGenBench extends existing benchmarks to many Indic languages
through human curation providing multi-way parallel evaluation data for many
under-represented Indic languages for the first time. We evaluate a wide range
of proprietary and open-source LLMs including GPT-3.5, GPT-4, PaLM-2, mT5,
Gemma, BLOOM and LLaMA on IndicGenBench in a variety of settings. The largest
PaLM-2 models performs the best on most tasks, however, there is a significant
performance gap in all languages compared to English showing that further
research is needed for the development of more inclusive multilingual language
models. IndicGenBench is released at
www.github.com/google-research-datasets/indic-gen-bench

印度是一个拥有 14 亿人口的语言多样性国家，为了促进多语言大型语言模型的研究和评估，我们发布了 IndicGenBench，这是一个覆盖 13 种脚本、4 种语系以及 29 种印度语言的最大基准。我们对包括 GPT-3.5、GPT-4、PaLM-2、mT5、Gemma、BLOOM 和 LLaMA 在内的各种大型语言模型在 IndicGenBench 上的表现进行了评估，结果显示多数任务中表现最好的是最大的 PaLM-2 模型，但与英语相比，所有语言都存在显著的性能差距，因此我们需要进一步的研究来开发更具包容性的多语言语言模型。