Failure detection in automated image classification is a critical safeguard
for clinical deployment. Detected failure cases can be referred to human
assessment, ensuring patient safety in computer-aided clinical decision making.
Despite its paramount importance, there is insufficient evidence about the
ability of state-of-the-art confidence scoring methods to detect test-time
failures of classification models in the context of medical imaging. This paper
provides a reality check, establishing the performance of in-domain
misclassification detection methods, benchmarking 9 widely used confidence
scores on 6 medical imaging datasets with different imaging modalities, in
multiclass and binary classification settings. Our experiments show that the
problem of failure detection is far from being solved. We found that none of
the benchmarked advanced methods proposed in the computer vision and machine
learning literature can consistently outperform a simple softmax baseline,
demonstrating that improved out-of-distribution detection or model calibration
do not necessarily translate to improved in-domain misclassification detection.
Our developed testbed facilitates future work in this important area

本文研究了自动图像分类中的故障检测，通过对 6 个医学影像数据集的测试发现，现有的置信度评分方法无法准确检测分类模型的测试失败，建议今后加强研究，提高故障检测的准确性。