Large Language Models (LLMs) are increasingly being used to generate
synthetic data for training and evaluating models. However, it is unclear
whether they can generate a good quality of question answering (QA) dataset
that incorporates knowledge and cultural nuance embedded in a language,
especially for low-resource languages. In this study, we investigate the
effectiveness of using LLMs in generating culturally relevant commonsense QA
datasets for Indonesian and Sundanese languages. To do so, we create datasets
for these languages using various methods involving both LLMs and human
annotators. Our experiments show that the current best-performing LLM, GPT-4
Turbo, is capable of generating questions with adequate knowledge in Indonesian
but not in Sundanese, highlighting the performance discrepancy between medium-
and lower-resource languages. We also benchmark various LLMs on our generated
datasets and find that they perform better on the LLM-generated datasets
compared to those created by humans.

使用大型语言模型生成问题回答数据集，尤其是针对低资源语言，存在领域知识和文化细微差别的困难；实验证明最佳的大型语言模型 GPT-4 Turbo 能够生成具备充分知识的印度尼西亚语问题，但在苏丹尼斯语上表现欠佳，凸显中资源语言与低资源语言之间的性能差异；此外，我们对各种大型语言模型在生成数据集上进行基准测试，结果表明它们在由语言模型生成的数据集上表现比人工创造的数据集更好。