We present Multi-EuP, a new multilingual benchmark dataset, comprising 22K
multi-lingual documents collected from the European Parliament, spanning 24
languages. This dataset is designed to investigate fairness in a multilingual
information retrieval (IR) context to analyze both language and demographic
bias in a ranking context. It boasts an authentic multilingual corpus,
featuring topics translated into all 24 languages, as well as cross-lingual
relevance judgments. Furthermore, it offers rich demographic information
associated with its documents, facilitating the study of demographic bias. We
report the effectiveness of Multi-EuP for benchmarking both monolingual and
multilingual IR. We also conduct a preliminary experiment on language bias
caused by the choice of tokenization strategy.

Multi-EuP 是一个多语言基准数据集，由来自欧洲议会的 22K 多语言文档组成，涵盖 24 种语言。该数据集旨在研究多语言信息检索 (IR) 语境中的公平性，分析排名语境中的语言和人口统计偏差。它拥有一个真实的多语言语料库，涵盖了 24 种语言的主题翻译，以及跨语言的相关性判断。此外，它还提供与文档相关的丰富人口统计信息，方便研究人口统计偏差。我们报告了 Multi-EuP 在单语和多语信息检索基准测试方面的有效性。我们还对由分词策略选择引起的语言偏差进行了初步实验。