Distribution shifts are common in real-world datasets and can affect the
performance and reliability of deep learning models. In this paper, we study
two types of distribution shifts: diversity shifts, which occur when test
samples exhibit patterns unseen during training, and correlation shifts, which
occur when test data present a different correlation between seen invariant and
spurious features. We propose an integrated protocol to analyze both types of
shifts using datasets where they co-exist in a controllable manner. Finally, we
apply our approach to a real-world classification problem of skin cancer
analysis, using out-of-distribution datasets and specialized bias annotations.
Our protocol reveals three findings: 1) Models learn and propagate correlation
shifts even with low-bias training; this poses a risk of accumulating and
combining unaccountable weak biases; 2) Models learn robust features in high-
and low-bias scenarios but use spurious ones if test samples have them; this
suggests that spurious correlations do not impair the learning of robust
features; 3) Diversity shift can reduce the reliance on spurious correlations;
this is counter intuitive since we expect biased models to depend more on
biases when invariant features are missing. Our work has implications for
distribution shift research and practice, providing new insights into how
models learn and rely on spurious correlations under different types of shifts.

本文针对分布转移对深度学习模型性能和可靠性的影响，研究了两种类型的分布转移：多样性转移和相关性转移，并提出了一种综合协议来分析这两种转移。应用我们的方法到一个真实的皮肤癌分类问题，最终有三个发现：模型在低偏差情况下仍学习和传播与转移相关的内容；模型学习了强健的特征但仍使用其他的特征；多样性转移可以降低偏差模型对于与转移相关的特征的依赖性。

即使是微小的相关性和多样性变化也会导致数据集偏差问题

Even Small Correlation and Diversity Shifts Pose Dataset-Bias Issues

Model fairness is an essential element for Trustworthy AI. While many
techniques for model fairness have been proposed, most of them assume that the
training and deployment data distributions are identical, which is often not
true in practice. In particular, when the bias between labels and sensitive
groups changes, the fairness of the trained model is directly influenced and
can worsen. We make two contributions for solving this problem. First, we
analytically show that existing in-processing fair algorithms have fundamental
limits in accuracy and group fairness. We introduce the notion of correlation
shifts, which can explicitly capture the change of the above bias. Second, we
propose a novel pre-processing step that samples the input data to reduce
correlation shifts and thus enables the in-processing approaches to overcome
their limitations. We formulate an optimization problem for adjusting the data
ratio among labels and sensitive groups to reflect the shifted correlation. A
key benefit of our approach lies in decoupling the roles of pre- and
in-processing approaches: correlation adjustment via pre-processing and
unfairness mitigation on the processed data via in-processing. Experiments show
that our framework effectively improves existing in-processing fair algorithms
w.r.t. accuracy and fairness, both on synthetic and real datasets.

本文提出了一种解决模型公平性中的相关性变化问题的新颖方法，即引入相关性偏移的概念和采用预处理步骤以减少相关性偏移，并借此通过优化问题来调整数据比率，从而使得数据处理算法可以有效地提高其准确性和公平性。