Real-world datasets may contain multiple features that explain the training
data equally well, i.e., learning any of them would lead to correct predictions
on the training data. However, many of them can be spurious, i.e., lose their
predictive power under a distribution shift and fail to generalize to
out-of-distribution (OOD) data. Recently developed ``diversification'' methods
approach this problem by finding multiple diverse hypotheses that rely on
different features. This paper aims to study this class of methods and identify
the key components contributing to their OOD generalization abilities.
We show that (1) diversification methods are highly sensitive to the
distribution of the unlabeled data used for diversification and can
underperform significantly when away from a method-specific sweet spot. (2)
Diversification alone is insufficient for OOD generalization. The choice of the
used learning algorithm, e.g., the model's architecture and pretraining, is
crucial, and using the second-best choice leads to an up to 20% absolute drop
in accuracy.(3) The optimal choice of learning algorithm depends on the
unlabeled data, and vice versa.Finally, we show that the above pitfalls cannot
be alleviated by increasing the number of diverse hypotheses, allegedly the
major feature of diversification methods.
These findings provide a clearer understanding of the critical design factors
influencing the OOD generalization of diversification methods. They can guide
practitioners in how to use the existing methods best and guide researchers in
developing new, better ones.

多样化方法对无标签数据的分布敏感，单纯多样化无法实现 OOD 泛化，学习算法的选择至关重要，将导致精度绝对下降 20％；选择最佳学习算法依赖于无标签数据，同时无标签数据也依赖于学习算法的选择。增加多样性假设数量不能解决这些问题。以上发现为了深入理解多样化方法的 OOD 泛化提供了关键设计因素，指导实践和研究。