In this study, we introduce BEnQA, a dataset comprising parallel Bengali and
English exam questions for middle and high school levels in Bangladesh. Our
dataset consists of approximately 5K questions covering several subjects in
science with different types of questions, including factual, application, and
reasoning-based questions. We benchmark several Large Language Models (LLMs)
with our parallel dataset and observe a notable performance disparity between
the models in Bengali and English. We also investigate some prompting methods,
and find that Chain-of-Thought prompting is beneficial mostly on reasoning
questions, but not so much on factual ones. We also find that appending English
translation helps to answer questions in Bengali. Our findings point to
promising future research directions for improving the performance of LLMs in
Bengali and more generally in low-resource languages.

本研究介绍了 BEnQA 数据集，该数据集包括孟加拉国中学和高中水平的平行孟加拉语和英语考试题目。我们的数据集包含大约 5,000 个问题，涵盖了科学的多个学科，并包含事实、应用和推理类问题。我们使用平行数据集对多个大型语言模型进行了基准测试，并观察到了在孟加拉语和英语中模型之间明显的性能差异。我们还研究了一些提示方法，并发现在推理问题上更有利的是 “思考链” 提示，而在事实问题上则不太有利。我们还发现，附加英语翻译有助于用孟加拉语回答问题。我们的研究结果指向了改善孟加拉语和其他资源有限语言中大型语言模型性能的有希望的未来研究方向。