The BBQ (Bias Benchmark for Question Answering) dataset enables the
evaluation of the social biases that language models (LMs) exhibit in
downstream tasks. However, it is challenging to adapt BBQ to languages other
than English as social biases are culturally dependent. In this paper, we
devise a process to construct a non-English bias benchmark dataset by
leveraging the English BBQ dataset in a culturally adaptive way and present the
KoBBQ dataset for evaluating biases in Question Answering (QA) tasks in Korean.
We identify samples from BBQ into three classes: Simply-Translated (can be used
directly after cultural translation), Target-Modified (requires localization in
target groups), and Sample-Removed (does not fit Korean culture). We further
enhance the cultural relevance to Korean culture by adding four new categories
of bias specific to Korean culture and newly creating samples based on Korean
literature. KoBBQ consists of 246 templates and 4,740 samples across 12
categories of social bias. Using KoBBQ, we measure the accuracy and bias scores
of several state-of-the-art multilingual LMs. We demonstrate the differences in
the bias of LMs in Korean and English, clarifying the need for hand-crafted
data considering cultural differences.

通过利用英文 BBQ 数据集以适应文化方式构建非英文偏见基准数据集，本文提出了 KoBBQ 数据集用于评估韩语问答任务中的偏见，并通过将韩国文化相关性增强到韩国文化，补充了四个韩国文化特定偏见类别，并基于韩国文学创建了新样本。使用 KoBBQ 数据集，衡量了几种最先进的多语言语言模型的准确性和偏见评分，展示了韩语和英语中语言模型的偏见差异，进一步阐明了考虑文化差异需要手工制作的数据的必要性。