This work aims to develop a measure that can accurately rank the performance
of various classifiers when they are tested on unlabeled data from
out-of-distribution (OOD) distributions. We commence by demonstrating that
conventional uncertainty metrics, notably the maximum Softmax prediction
probability, possess inherent utility in forecasting model generalization
across certain OOD contexts. Building on this insight, we introduce a new
measure called Softmax Correlation (SoftmaxCorr). It calculates the cosine
similarity between a class-class correlation matrix, constructed from Softmax
output vectors across an unlabeled test dataset, and a predefined reference
matrix that embodies ideal class correlations. A high resemblance of
predictions to the reference matrix signals that the model delivers confident
and uniform predictions across all categories, reflecting minimal uncertainty
and confusion. Through rigorous evaluation across a suite of datasets,
including ImageNet, CIFAR-10, and WILDS, we affirm the predictive validity of
SoftmaxCorr in accurately forecasting model performance within both
in-distribution (ID) and OOD settings. Furthermore, we discuss the limitations
of our proposed measure and suggest avenues for future research.

本研究旨在发展一种能准确评估各种分类器在来自未标记数据的离域分布上进行测试时的性能排名的度量方法。通过展示常规的不确定性度量方法，特别是最大 Softmax 预测概率，在某些离域环境下预测模型泛化的内在效用，我们首先介绍了一种名为 SoftmaxCorr 的新度量方法。它计算了一个由 Softmax 输出向量构建的类 - 类相关矩阵与一个预定义的具有理想类别相关性的参考矩阵之间的余弦相似度。如果预测与参考矩阵相似度很高，则表明模型在所有类别上提供自信和统一的预测，反映出最小的不确定性和混淆。通过对一系列数据集的严格评估，包括 ImageNet、CIFAR-10 和 WILDS，我们确认了 SoftmaxCorr 在准确预测模型性能方面在分布内和离域设置中的有效性。此外，我们讨论了我们提出的度量方法的局限性，并提出了未来研究的方向。