Recent work has shown that the performance of machine learning models can
vary substantially when models are evaluated on data drawn from a distribution
that is close to but different from the training distribution. As a result,
predicting model performance on unseen distributions is an important challenge.
Our work connects techniques from domain adaptation and predictive uncertainty
literature, and allows us to predict model accuracy on challenging unseen
distributions without access to labeled data. In the context of distribution
shift, distributional distances are often used to adapt models and improve
their performance on new domains, however accuracy estimation, or other forms
of predictive uncertainty, are often neglected in these investigations. Through
investigating a wide range of established distributional distances, such as
Frechet distance or Maximum Mean Discrepancy, we determine that they fail to
induce reliable estimates of performance under distribution shift. On the other
hand, we find that the difference of confidences (DoC) of a classifier's
predictions successfully estimates the classifier's performance change over a
variety of shifts. We specifically investigate the distinction between
synthetic and natural distribution shifts and observe that despite its
simplicity DoC consistently outperforms other quantifications of distributional
difference. $DoC$ reduces predictive error by almost half ($46\%$) on several
realistic and challenging distribution shifts, e.g., on the ImageNet-Vid-Robust
and ImageNet-Rendition datasets.

研究发现现有的分布距离度量方法，例如 Frechetdistance 或 Maximum Mean Discrepancy，不能在分布偏移中可靠地估计模型性能；然而，分类器预测的置信度差异能够成功地估计分类器在各种转移情况下的性能变化，并且在几个现实和具有挑战性的分布转移中有效地减少预测误差近一半（46％），这一方法称为 DoC (Difference of Confidences)。

对未见过分布进行有信心的预测

Predicting with Confidence on Unseen Distributions

Recovering the occlusion relationships between objects is a fundamental human
visual ability which yields important information about the 3D world. In this
paper we propose a deep network architecture, called DOC, which acts on a
single image, detects object boundaries and estimates the border ownership
(i.e. which side of the boundary is foreground and which is background). We
represent occlusion relations by a binary edge map, to indicate the object
boundary, and an occlusion orientation variable which is tangential to the
boundary and whose direction specifies border ownership by a left-hand rule. We
train two related deep convolutional neural networks, called DOC, which exploit
local and non-local image cues to estimate this representation and hence
recover occlusion relations. In order to train and test DOC we construct a
large-scale instance occlusion boundary dataset using PASCAL VOC images, which
we call the PASCAL instance occlusion dataset (PIOD). This contains 10,000
images and hence is two orders of magnitude larger than existing occlusion
datasets for outdoor images. We test two variants of DOC on PIOD and on the
BSDS occlusion dataset and show they outperform state-of-the-art methods.
Finally, we perform numerous experiments investigating multiple settings of DOC
and transfer between BSDS and PIOD, which provides more insights for further
study of occlusion estimation.

本文提出了一种名为 DOC 的深度网络架构，它能在单个图像上检测物体边界并估计边界所有权，可用于恢复遮挡关系，并通过大规模实例遮挡边界数据集 PIOD 进行了验证。