Collaborative competitions have gained popularity in the scientific and
technological fields. These competitions involve defining tasks, selecting
evaluation scores, and devising result verification methods. In the standard
scenario, participants receive a training set and are expected to provide a
solution for a held-out dataset kept by organizers. An essential challenge for
organizers arises when comparing algorithms' performance, assessing multiple
participants, and ranking them. Statistical tools are often used for this
purpose; however, traditional statistical methods often fail to capture
decisive differences between systems' performance. This manuscript describes an
evaluation methodology for statistically analyzing competition results and
competition. The methodology is designed to be universally applicable; however,
it is illustrated using eight natural language competitions as case studies
involving classification and regression problems. The proposed methodology
offers several advantages, including off-the-shell comparisons with correction
mechanisms and the inclusion of confidence intervals. Furthermore, we introduce
metrics that allow organizers to assess the difficulty of competitions. Our
analysis shows the potential usefulness of our methodology for effectively
evaluating competition results.

合作竞争的科学和技术领域变得越来越受欢迎。本文描述了一种评估方法来对比竞赛结果和竞争。这种方法具有普适性，但是以八个自然语言竞赛为案例进行了说明，涉及分类和回归问题。所提出的方法具有多种优势，包括与修正机制的即插即用比较和置信区间的包含。此外，我们引入了一些指标，使组织者能够评估竞赛的难度。我们的分析显示了我们方法在有效评估竞赛结果方面的潜在有用性。

自然语言处理竞赛中系统性能分析

Analysis of Systems' Performance in Natural Language Processing  Competitions

Solver competitions play a prominent role in assessing and advancing the
state of the art for solving many problems in AI and beyond. Notably, in many
areas of AI, competitions have had substantial impact in guiding research and
applications for many years, and for a solver to be ranked highly in a
competition carries considerable weight. But to which extent can we expect
competition results to generalise to sets of problem instances different from
those used in a particular competition? This is the question we investigate
here, using statistical resampling techniques. We show that the rankings
resulting from the standard interpretation of competition results can be very
sensitive to even minor changes in the benchmark instance set used as the basis
for assessment and can therefore not be expected to carry over to other samples
from the same underlying instance distribution. To address this problem, we
introduce a novel approach to statistically meaningful analysis of competition
results based on resampling performance data. Our approach produces confidence
intervals of competition scores as well as statistically robust solver rankings
with bounded error. Applied to recent SAT, AI planning and computer vision
competitions, our analysis reveals frequent statistical ties in solver
performance as well as some inversions of ranks compared to the official
results based on simple scoring.

通过统计重采样技术，本研究调查了竞赛结果的普遍适用性问题，并提出了一种基于重采样性能数据的统计稳健解算器排名方法。应用于最近的 SAT、AI 规划和计算机视觉竞赛，分析结果显示了解算器性能的频繁统计并列以及与基于简单评分的官方结果相比的排名倒置。