Ordinal classification models assign higher penalties to predictions further
away from the true class. As a result, they are appropriate for relevant
diagnostic tasks like disease progression prediction or medical image grading.
The consensus for assessing their categorical predictions dictates the use of
distance-sensitive metrics like the Quadratic-Weighted Kappa score or the
Expected Cost. However, there has been little discussion regarding how to
measure performance of probabilistic predictions for ordinal classifiers. In
conventional classification, common measures for probabilistic predictions are
Proper Scoring Rules (PSR) like the Brier score, or Calibration Errors like the
ECE, yet these are not optimal choices for ordinal classification. A PSR named
Ranked Probability Score (RPS), widely popular in the forecasting field, is
more suitable for this task, but it has received no attention in the image
analysis community. This paper advocates the use of the RPS for image grading
tasks. In addition, we demonstrate a counter-intuitive and questionable
behavior of this score, and propose a simple fix for it. Comprehensive
experiments on four large-scale biomedical image grading problems over three
different datasets show that the RPS is a more suitable performance metric for
probabilistic ordinal predictions. Code to reproduce our experiments can be
found at this https URL .

本文提出了使用排序概率得分（RPS）作为医学图像分级任务的性能度量指标，并展示了该得分的反直觉和可疑行为，并提出了简单的修复方法。通过对四个大规模生物医学图像分级问题进行全面实验，证明了 RPS 是适合概率排序预测的性能度量指标。