Evaluation in NLP is usually done by comparing the scores of competing
systems independently averaged over a common set of test instances. In this
work, we question the use of averages for aggregating evaluation scores into a
final number used to decide which system is best, since the average, as well as
alternatives such as the median, ignores the pairing arising from the fact that
systems are evaluated on the same test instances. We illustrate the importance
of taking the instance-level pairing of evaluation scores into account and
demonstrate, both theoretically and empirically, the advantages of aggregation
methods based on pairwise comparisons, such as the Bradley-Terry (BT) model, a
mechanism based on the estimated probability that a given system scores better
than another on the test set. By re-evaluating 296 real NLP evaluation setups
across four tasks and 18 evaluation metrics, we show that the choice of
aggregation mechanism matters and yields different conclusions as to which
systems are state of the art in about 30% of the setups. To facilitate the
adoption of pairwise evaluation, we release a practical tool for performing the
full analysis of evaluation scores with the mean, median, BT, and two variants
of BT (Elo and TrueSkill), alongside functionality for appropriate statistical
testing.

本 paper 探讨了语言处理领域中评估模型常使用的平均值等聚合方法存在的缺陷，并通过理论与实验证明采用基于对比的聚合方法如测量模型间比较的 Bradley-Terry（BT）模型可以更好的反映模型之间的差异。最后，研究提供了一个可供实用的工具来辅助基于 BT 方法进行实验评估。