Representation learning has significantly driven the field to develop
pretrained models that can act as a valuable starting point when transferring
to new datasets. With the rising demand for reliable machine learning and
uncertainty quantification, there is a need for pretrained models that not only
provide embeddings but also transferable uncertainty estimates. To guide the
development of such models, we propose the Uncertainty-aware Representation
Learning (URL) benchmark. Besides the transferability of the representations,
it also measures the zero-shot transferability of the uncertainty estimate
using a novel metric. We apply URL to evaluate eleven uncertainty quantifiers
that are pretrained on ImageNet and transferred to eight downstream datasets.
We find that approaches that focus on the uncertainty of the representation
itself or estimate the prediction risk directly outperform those that are based
on the probabilities of upstream classes. Yet, achieving transferable
uncertainty quantification remains an open challenge. Our findings indicate
that it is not necessarily in conflict with traditional representation learning
goals. Code is provided under this https URL .

基于预训练模型的不确定性感知表示学习（URL）基准评估了 11 种预训练模型在从 ImageNet 转移到八个下游数据集时的表示的可迁移性和不确定性的可迁移性，表明关注表示本身的不确定性或直接估计预测风险的方法优于基于上游类别概率的方法，但实现可迁移的不确定性量化仍然是一个挑战。