In Machine Learning, a benchmark refers to an ensemble of datasets associated
with one or multiple metrics together with a way to aggregate different systems
performances. They are instrumental in (i) assessing the progress of new
methods along different axes and (ii) selecting the best systems for practical
use. This is particularly the case for NLP with the development of large
pre-trained models (e.g. GPT, BERT) that are expected to generalize well on a
variety of tasks. While the community mainly focused on developing new datasets
and metrics, there has been little interest in the aggregation procedure, which
is often reduced to a simple average over various performance measures.
However, this procedure can be problematic when the metrics are on a different
scale, which may lead to spurious conclusions. This paper proposes a new
procedure to rank systems based on their performance across different tasks.
Motivated by the social choice theory, the final system ordering is obtained
through aggregating the rankings induced by each task and is theoretically
grounded. We conduct extensive numerical experiments (on over 270k scores) to
assess the soundness of our approach both on synthetic and real scores (e.g.
GLUE, EXTREM, SEVAL, TAC, FLICKR). In particular, we show that our method
yields different conclusions on state-of-the-art systems than the
mean-aggregation procedure while being both more reliable and robust.

该研究提出一种新的基于社会选择理论的系统排名方法，用于评估机器学习中使用 benchmark 的效果。该方法在评估自然语言处理中的预训练模型时更为可靠且稳健，有别于简单的平均聚合，可为实际应用提供最佳的系统选择。