Collaborative competitions have gained popularity in the scientific and
technological fields. These competitions involve defining tasks, selecting
evaluation scores, and devising result verification methods. In the standard
scenario, participants receive a training set and are expected to provide a
solution for a held-out dataset kept by organizers. An essential challenge for
organizers arises when comparing algorithms' performance, assessing multiple
participants, and ranking them. Statistical tools are often used for this
purpose; however, traditional statistical methods often fail to capture
decisive differences between systems' performance. This manuscript describes an
evaluation methodology for statistically analyzing competition results and
competition. The methodology is designed to be universally applicable; however,
it is illustrated using eight natural language competitions as case studies
involving classification and regression problems. The proposed methodology
offers several advantages, including off-the-shell comparisons with correction
mechanisms and the inclusion of confidence intervals. Furthermore, we introduce
metrics that allow organizers to assess the difficulty of competitions. Our
analysis shows the potential usefulness of our methodology for effectively
evaluating competition results.

合作竞争的科学和技术领域变得越来越受欢迎。本文描述了一种评估方法来对比竞赛结果和竞争。这种方法具有普适性，但是以八个自然语言竞赛为案例进行了说明，涉及分类和回归问题。所提出的方法具有多种优势，包括与修正机制的即插即用比较和置信区间的包含。此外，我们引入了一些指标，使组织者能够评估竞赛的难度。我们的分析显示了我们方法在有效评估竞赛结果方面的潜在有用性。