AI tools are increasingly deployed in community contexts. However, datasets
used to evaluate AI are typically created by developers and annotators outside
a given community, which can yield misleading conclusions about AI performance.
How might we empower communities to drive the intentional design and curation
of evaluation datasets for AI that impacts them? We investigate this question
on Wikipedia, an online community with multiple AI-based content moderation
tools deployed. We introduce Wikibench, a system that enables communities to
collaboratively curate AI evaluation datasets, while navigating ambiguities and
differences in perspective through discussion. A field study on Wikipedia shows
that datasets curated using Wikibench can effectively capture community
consensus, disagreement, and uncertainty. Furthermore, study participants used
Wikibench to shape the overall data curation process, including refining label
definitions, determining data inclusion criteria, and authoring data
statements. Based on our findings, we propose future directions for systems
that support community-driven data curation.

AI 工具越来越多地在社区环境中部署。然而，用于评估 AI 的数据集通常由社区之外的开发者和注释者创建，这可能对 AI 性能产生误导性结论。本研究调查了如何赋予社区推动 AI 评估数据集的有意设计和策划的能力，我们在维基百科进行了探索。我们引入了 Wikibench，这是一个系统，可以让社区协作策划 AI 评估数据集，并通过讨论解决歧义和观点差异。维基百科的实地研究表明，使用 Wikibench 策划的数据集能够有效捕捉到社区的共识、分歧和不确定性。此外，研究参与者使用 Wikibench 来塑造整个数据策划过程，包括改善标签定义、确定数据包含标准和撰写数据说明。根据我们的研究结果，我们提出了支持社区驱动的数据策划的未来发展方向。