Computer vision applications predict on digital images acquired by a camera
from physical scenes through light. However, conventional robustness benchmarks
rely on perturbations in digitized images, diverging from distribution shifts
occurring in the image acquisition process. To bridge this gap, we introduce a
new distribution shift dataset, ImageNet-ES, comprising variations in
environmental and camera sensor factors by directly capturing 202k images with
a real camera in a controllable testbed. With the new dataset, we evaluate
out-of-distribution (OOD) detection and model robustness. We find that existing
OOD detection methods do not cope with the covariate shifts in ImageNet-ES,
implying that the definition and detection of OOD should be revisited to
embrace real-world distribution shifts. We also observe that the model becomes
more robust in both ImageNet-C and -ES by learning environment and sensor
variations in addition to existing digital augmentations. Lastly, our results
suggest that effective shift mitigation via camera sensor control can
significantly improve performance without increasing model size. With these
findings, our benchmark may aid future research on robustness, OOD, and camera
sensor control for computer vision. Our code and dataset are available at
this https URL

计算机视觉应用中，现有的稳健性基准测试依赖于对数字图像的扰动，从而偏离了图像获取过程中发生的分布转变。为了弥合这一差距，我们引入了一个新的分布转变数据集 ImageNet-ES，该数据集由一个真实相机在一个可控实验环境中直接捕获的 202k 张图像组成。通过这个新数据集，我们评估了超出分布范围 (Out-of-Distribution, OOD) 检测和模型稳健性。我们发现，现有的 OOD 检测方法无法应对 ImageNet-ES 中的协变量转变，这意味着应重新审视 OOD 的定义和检测，以适应真实世界的分布变化。我们还观察到，通过学习环境和传感器的变化，模型在 ImageNet-C 和 - ES 方面都更加稳健，这是在现有数字增强的基础上的补充。最后，我们的结果表明，通过相机传感器控制有效地减轻转变可以显著提高性能，而无需增加模型大小。通过这些发现，我们的基准测试可能有助于未来关于计算机视觉稳健性、OOD 和相机传感器控制的研究。我们的代码和数据集可在此 https URL 找到。