Although commonsense reasoning is greatly shaped by cultural and geographical
factors, previous studies on language models have predominantly centered on
English cultures, potentially resulting in an Anglocentric bias. In this paper,
we introduce IndoCulture, aimed at understanding the influence of geographical
factors on language model reasoning ability, with a specific emphasis on the
diverse cultures found within eleven Indonesian provinces. In contrast to prior
works that relied on templates (Yin et al., 2022) and online scrapping (Fung et
al., 2024), we created IndoCulture by asking local people to manually develop
the context and plausible options based on predefined topics. Evaluations of 23
language models reveal several insights: (1) even the best open-source model
struggles with an accuracy of 53.2%, (2) models often provide more accurate
predictions for specific provinces, such as Bali and West Java, and (3) the
inclusion of location contexts enhances performance, especially in larger
models like GPT-4, emphasizing the significance of geographical context in
commonsense reasoning.

本研究旨在了解地理因素对语言模型的推理能力的影响，以特别强调印尼十一个不同省份内的多元文化。评估 23 种语言模型表明：（1）即使是最优秀的开源模型，准确率也只有 53.2%；（2）模型在某些特定省份（如巴厘岛和西爪哇）的预测通常更准确；（3）地理背景的加入提升了性能，尤其是在像 GPT-4 这样的大型模型中，强调了地理背景在常识推理中的重要性。