Frontier large language models (LLMs) are developed by researchers and
practitioners with skewed cultural backgrounds and on datasets with skewed
sources. However, LLMs' (lack of) multicultural knowledge cannot be effectively
assessed with current methods for developing benchmarks. Existing multicultural
evaluations primarily rely on expensive and restricted human annotations or
potentially outdated internet resources. Thus, they struggle to capture the
intricacy, dynamics, and diversity of cultural norms. LLM-generated benchmarks
are promising, yet risk propagating the same biases they are meant to measure.
To synergize the creativity and expert cultural knowledge of human annotators
and the scalability and standardizability of LLM-based automation, we introduce
CulturalTeaming, an interactive red-teaming system that leverages human-AI
collaboration to build truly challenging evaluation dataset for assessing the
multicultural knowledge of LLMs, while improving annotators' capabilities and
experiences. Our study reveals that CulturalTeaming's various modes of AI
assistance support annotators in creating cultural questions, that modern LLMs
fail at, in a gamified manner. Importantly, the increased level of AI
assistance (e.g., LLM-generated revision hints) empowers users to create more
difficult questions with enhanced perceived creativity of themselves, shedding
light on the promises of involving heavier AI assistance in modern evaluation
dataset creation procedures. Through a series of 1-hour workshop sessions, we
gather CULTURALBENCH-V0.1, a compact yet high-quality evaluation dataset with
users' red-teaming attempts, that different families of modern LLMs perform
with accuracy ranging from 37.7% to 72.2%, revealing a notable gap in LLMs'
multicultural proficiency.

CulturalTeaming 是一个人工智能系统，与人类合作创建多元文化知识评估数据集，通过与人类的互动合作，利用大型语言模型的自动化功能，提高文本注释者的能力并改善他们的体验，从而评估大型语言模型的多元文化知识，并揭示了现代大型语言模型在多元文化能力上存在的显著差距。