While measuring bias and robustness in coreference resolution are important goals, such measurements are only as good as the tools we use to measure them with. Winogender schemas (Rudinger et al., 2018) are an influential dataset proposed to evaluate gender bias in coreference resolution, but a closer look reveals issues with the data that compromise its use for reliable evaluation, including treating different pronominal forms as equivalent, violations of template constraints, and typographical errors. We identify these issues and fix them, contributing a new dataset: WinoPron. Our changes affect performance with state-of-the-art supervised coreference resolution systems as well as all model sizes of the language model FLAN-T5, with F1 dropping on average 10 percentage points. We also propose a new method to evaluate pronominal bias in coreference resolution that goes beyond the binary. With this method and our new dataset which is balanced for grammatical case, we empirically demonstrate that bias characteristics vary not just across pronoun sets, but also across surface forms of those sets.

本研究针对使用 Winogender 模式评估核心指代解析中的性别偏见时存在的数据问题进行了深入分析，包括代词形式的等价处理、模板约束违反及排版错误。我们提出了新数据集 WinoPron，并开发了一种新的评估方法，以更全面地衡量核心指代解析中的代名词偏见，发现偏见特征不仅因代词集而异，也因其表面形式而异。

WinoPron：重新审视英语 Winogender 模式的一致性、覆盖范围和语法案例