Large language models (LLMs) are commonly used for long-form question
answering, which requires them to generate paragraph-length answers to complex
questions. While long-form QA has been well-studied in English via many
different datasets and evaluation metrics, this research has not been extended
to cover most other languages. To bridge this gap, we introduce CaLMQA, a
collection of 2.6K complex questions spanning 23 languages, including
under-resourced, rarely-studied languages such as Fijian and Kirundi. Our
dataset includes both naturally-occurring questions collected from community
web forums as well as questions written by native speakers, whom we hire for
this purpose. Our process yields diverse, complex questions that reflect
cultural topics (e.g. traditions, laws, news) and the language usage of native
speakers. We conduct automatic evaluation across a suite of open- and
closed-source models using our novel metric CaLMScore, which detects incorrect
language and token repetitions in answers, and observe that the quality of
LLM-generated answers degrades significantly for some low-resource languages.
We perform human evaluation on a subset of models and see that model
performance is significantly worse for culturally specific questions than for
culturally agnostic questions. Our findings highlight the need for further
research in LLM multilingual capabilities and non-English LFQA evaluation.

通过介绍 CaLMQA，一个涵盖 23 种语言的 2.6K 多样化问题集，我们发现大型语言模型在生成复杂问题的长篇答案时在某些低资源语言方面的质量明显下降，这突显了 LLM 多语言能力和非英语长篇问题回答评估中进一步研究的需求。