While clinical trials are the state-of-the-art methods to assess the effect
of new medication in a comparative manner, benchmarking in the field of medical
image analysis is performed by so-called challenges. Recently, comprehensive
analysis of multiple biomedical image analysis challenges revealed large
discrepancies between the impact of challenges and quality control of the
design and reporting standard. This work aims to follow up on these results and
attempts to address the specific question of the reproducibility of the
participants methods. In an effort to determine whether alternative
interpretations of the method description may change the challenge ranking, we
reproduced the algorithms submitted to the 2019 Robust Medical Image
Segmentation Challenge (ROBUST-MIS). The leaderboard differed substantially
between the original challenge and reimplementation, indicating that challenge
rankings may not be sufficiently reproducible.

采用临床试验评估新药疗效的同时，对医学图像分析领域进行基准测试需要通过挑战赛，然而多个医学图像分析挑战赛的分析表明，其质量控制标准可能存在巨大差异，因此本文针对参与者方法的可复现性问题提出了一个观点，尝试寻找替代性解释并重现算法提交结果，结果表明挑战赛排名可能不够可靠。

挑战结果不可重复

Challenge Results Are Not Reproducible

Biomedical challenges have become the de facto standard for benchmarking
biomedical image analysis algorithms. While the number of challenges is
steadily increasing, surprisingly little effort has been invested in ensuring
high quality design, execution and reporting for these international
competitions. Specifically, results analysis and visualization in the event of
uncertainties have been given almost no attention in the literature. Given
these shortcomings, the contribution of this paper is two-fold: (1) We present
a set of methods to comprehensively analyze and visualize the results of
single-task and multi-task challenges and apply them to a number of simulated
and real-life challenges to demonstrate their specific strengths and
weaknesses; (2) We release the open-source framework challengeR as part of this
work to enable fast and wide adoption of the methodology proposed in this
paper. Our approach offers an intuitive way to gain important insights into the
relative and absolute performance of algorithms, which cannot be revealed by
commonly applied visualization techniques. This is demonstrated by the
experiments performed within this work. Our framework could thus become an
important tool for analyzing and visualizing challenge results in the field of
biomedical image analysis and beyond.

本文提出了一种基于 challengeR 开源框架的集成分析和可视化方法，以了解算法的相对和绝对性能，并证明该方法在医学图像分析领域和其他领域中具有重要应用价值。