Each new generation of English-oriented Large Language Models (LLMs) exhibits
enhanced cross-lingual transfer capabilities and significantly outperforms
older LLMs on low-resource languages. This prompts the question: Is there a
need for LLMs dedicated to a particular low-resource language? We aim to
explore this question for Bengali, a low-to-moderate resource Indo-Aryan
language native to the Bengal region of South Asia.
We compare the performance of open-weight and closed-source LLMs such as
LLaMA-3 and GPT-4 against fine-tuned encoder-decoder models across a diverse
set of Bengali downstream tasks, including translation, summarization,
paraphrasing, question-answering, and natural language inference. Our findings
reveal that while LLMs generally excel in reasoning tasks, their performance in
tasks requiring Bengali script generation is inconsistent. Key challenges
include inefficient tokenization of Bengali script by existing LLMs, leading to
increased computational costs and potential performance degradation.
Additionally, we highlight biases in machine-translated datasets commonly used
for Bengali NLP tasks. We conclude that there is a significant need for a
Bengali-oriented LLM, but the field currently lacks the high-quality
pretraining and instruction-tuning datasets necessary to develop a highly
effective model.

当前需要面临的问题是，是否需要专门面向低资源语言的大型语言模型（LLMs）？我们通过比较英语导向的 LLMs 与细调编码解码模型在孟加拉语下游任务中的表现，发现现有 LLMs 在生成孟加拉语脚本方面面临着挑战，同时也存在着一些数据集方面的偏见，因此需要开发更高效的孟加拉语导向 LLM。