The rapid development of Large Language Models (LLMs) and the emergence of
novel abilities with scale have necessitated the construction of holistic,
diverse and challenging benchmarks such as HELM and BIG-bench. However, at the
moment, most of these benchmarks focus only on performance in English and
evaluations that include Southeast Asian (SEA) languages are few in number. We
therefore propose BHASA, a holistic linguistic and cultural evaluation suite
for LLMs in SEA languages. It comprises three components: (1) a NLP benchmark
covering eight tasks across Natural Language Understanding (NLU), Generation
(NLG) and Reasoning (NLR) tasks, (2) LINDSEA, a linguistic diagnostic toolkit
that spans the gamut of linguistic phenomena including syntax, semantics and
pragmatics, and (3) a cultural diagnostics dataset that probes for both
cultural representation and sensitivity. For this preliminary effort, we
implement the NLP benchmark only for Indonesian, Vietnamese, Thai and Tamil,
and we only include Indonesian and Tamil for LINDSEA and the cultural
diagnostics dataset. As GPT-4 is purportedly one of the best-performing
multilingual LLMs at the moment, we use it as a yardstick to gauge the
capabilities of LLMs in the context of SEA languages. Our initial experiments
on GPT-4 with BHASA find it lacking in various aspects of linguistic
capabilities, cultural representation and sensitivity in the targeted SEA
languages. BHASA is a work in progress and will continue to be improved and
expanded in the future.

在东南亚语言方面，BHASA 提供了一个综合的语言和文化评估套件，其中包括自然语言理解、生成和推理等任务的 NLP 基准、语言诊断工具包 LINDSEA 以及文化诊断数据集，初步实验发现 GPT-4 在东南亚语言方面的语言能力、文化表达和敏感性等方面存在不足，BHASA 将在未来继续改进和扩展。