Deep neural networks have shown impressive performance for image-based
disease detection. Performance is commonly evaluated through clinical
validation on independent test sets to demonstrate clinically acceptable
accuracy. Reporting good performance metrics on test sets, however, is not
always a sufficient indication of the generalizability and robustness of an
algorithm. In particular, when the test data is drawn from the same
distribution as the training data, the iid test set performance can be an
unreliable estimate of the accuracy on new data. In this paper, we employ
stress testing to assess model robustness and subgroup performance disparities
in disease detection models. We design progressive stress testing using five
different bidirectional and unidirectional image perturbations with six
different severity levels. As a use case, we apply stress tests to measure the
robustness of disease detection models for chest X-ray and skin lesion images,
and demonstrate the importance of studying class and domain-specific model
behaviour. Our experiments indicate that some models may yield more robust and
equitable performance than others. We also find that pretraining
characteristics play an important role in downstream robustness. We conclude
that progressive stress testing is a viable and important tool and should
become standard practice in the clinical validation of image-based disease
detection models.

通过深度神经网络对基于图像的疾病检测进行研究并进行临床验证，通过应力测试评估模型的稳健性和亚组性能差异，发现某些模型能够产生更稳健和公正的性能，并且预训练特征对下游稳健性起重要作用，强调应力测试在图像疾病检测模型的临床验证中应成为标准实践。

医学图像分类中的鲁棒性压力测试

Robustness Stress Testing in Medical Image Classification

Informally, a 'spurious correlation' is the dependence of a model on some
aspect of the input data that an analyst thinks shouldn't matter. In machine
learning, these have a know-it-when-you-see-it character; e.g., changing the
gender of a sentence's subject changes a sentiment predictor's output. To check
for spurious correlations, we can 'stress test' models by perturbing irrelevant
parts of input data and seeing if model predictions change. In this paper, we
study stress testing using the tools of causal inference. We introduce
counterfactual invariance as a formalization of the requirement that changing
irrelevant parts of the input shouldn't change model predictions. We connect
counterfactual invariance to out-of-domain model performance, and provide
practical schemes for learning (approximately) counterfactual invariant
predictors (without access to counterfactual examples). It turns out that both
the means and implications of counterfactual invariance depend fundamentally on
the true underlying causal structure of the data -- in particular, whether the
label causes the features or the features cause the label. Distinct causal
structures require distinct regularization schemes to induce counterfactual
invariance. Similarly, counterfactual invariance implies different domain shift
guarantees depending on the underlying causal structure. This theory is
supported by empirical results on text classification.

本文通过对原因推断的工具进行研究，介绍了反事实不变性作为对无关输入部分进行更改时不应改变模型预测的要求的一个形式化。同时，提供了学习（近似）反事实不变预测器的实用方案，并对文本分类进行了实证研究。