Computer vision applications predict on digital images acquired by a camera
from physical scenes through light. However, conventional robustness benchmarks
rely on perturbations in digitized images, diverging from distribution shifts
occurring in the image acquisition process. To bridge this gap, we introduce a
new distribution shift dataset, ImageNet-ES, comprising variations in
environmental and camera sensor factors by directly capturing 202k images with
a real camera in a controllable testbed. With the new dataset, we evaluate
out-of-distribution (OOD) detection and model robustness. We find that existing
OOD detection methods do not cope with the covariate shifts in ImageNet-ES,
implying that the definition and detection of OOD should be revisited to
embrace real-world distribution shifts. We also observe that the model becomes
more robust in both ImageNet-C and -ES by learning environment and sensor
variations in addition to existing digital augmentations. Lastly, our results
suggest that effective shift mitigation via camera sensor control can
significantly improve performance without increasing model size. With these
findings, our benchmark may aid future research on robustness, OOD, and camera
sensor control for computer vision. Our code and dataset are available at
this https URL

计算机视觉应用中，现有的稳健性基准测试依赖于对数字图像的扰动，从而偏离了图像获取过程中发生的分布转变。为了弥合这一差距，我们引入了一个新的分布转变数据集 ImageNet-ES，该数据集由一个真实相机在一个可控实验环境中直接捕获的 202k 张图像组成。通过这个新数据集，我们评估了超出分布范围 (Out-of-Distribution, OOD) 检测和模型稳健性。我们发现，现有的 OOD 检测方法无法应对 ImageNet-ES 中的协变量转变，这意味着应重新审视 OOD 的定义和检测，以适应真实世界的分布变化。我们还观察到，通过学习环境和传感器的变化，模型在 ImageNet-C 和 - ES 方面都更加稳健，这是在现有数字增强的基础上的补充。最后，我们的结果表明，通过相机传感器控制有效地减轻转变可以显著提高性能，而无需增加模型大小。通过这些发现，我们的基准测试可能有助于未来关于计算机视觉稳健性、OOD 和相机传感器控制的研究。我们的代码和数据集可在此 https URL 找到。

环境和传感器领域的鲁棒性和超出分布的未探索面：协变量转换

Unexplored Faces of Robustness and Out-of-Distribution: Covariate Shifts  in Environment and Sensor Domains

Enhancing the robustness of vision algorithms in real-world scenarios is
challenging. One reason is that existing robustness benchmarks are limited, as
they either rely on synthetic data or ignore the effects of individual nuisance
factors. We introduce OOD-CV, a benchmark dataset that includes
out-of-distribution examples of 10 object categories in terms of pose, shape,
texture, context and the weather conditions, and enables benchmarking models
for image classification, object detection, and 3D pose estimation. In addition
to this novel dataset, we contribute extensive experiments using popular
baseline methods, which reveal that: 1. Some nuisance factors have a much
stronger negative effect on the performance compared to others, also depending
on the vision task. 2. Current approaches to enhance robustness have only
marginal effects, and can even reduce robustness. 3. We do not observe
significant differences between convolutional and transformer architectures. We
believe our dataset provides a rich testbed to study robustness and will help
push forward research in this area.

本文引入 OOD-CV 数据集，并发现某些干扰因素有更强烈的负面影响；当前的提高鲁棒性的方法只有微弱的效果，甚至可能削弱鲁棒性；我们的数据集提供了一个丰富的测试床，以研究鲁棒性，并有助于推进这一领域的研究。

OOD-CV: 自然图像个体干扰鲁棒性的离群分布偏移基准

OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of  Individual Nuisances in Natural Images

We present a combined scaling method - named BASIC - that achieves 85.7%
top-1 accuracy on the ImageNet ILSVRC-2012 validation set without learning from
any labeled ImageNet example. This accuracy surpasses best published similar
models - CLIP and ALIGN - by 9.3%. Our BASIC model also shows significant
improvements in robustness benchmarks. For instance, on 5 test sets with
natural distribution shifts such as ImageNet-{A,R,V2,Sketch} and ObjectNet, our
model achieves 84.3% top-1 average accuracy, only a small drop from its
original ImageNet accuracy. To achieve these results, we scale up the
contrastive learning framework of CLIP and ALIGN in three dimensions: data
size, model size, and batch size. Our dataset has 6.6B noisy image-text pairs,
which is 4x larger than ALIGN, and 16x larger than CLIP. Our largest model has
3B weights, which is 3.75x larger in parameters and 8x larger in FLOPs than
ALIGN and CLIP. Finally, our batch size is 65536 which is 2x more than CLIP and
4x more than ALIGN. We encountered two main challenges with the scaling rules
of BASIC. First, the main challenge with implementing the combined scaling
rules of BASIC is the limited memory of accelerators, such as GPUs and TPUs. To
overcome the memory limit, we propose two simple methods which make use of
gradient checkpointing and model parallelism. Second, while increasing the
dataset size and the model size has been the defacto method to improve the
performance of deep learning models like BASIC, the effect of a large
contrastive batch size on such contrastive-trained image-text models is not
well-understood. To shed light on the benefits of large contrastive batch
sizes, we develop a theoretical framework which shows that larger contrastive
batch sizes lead to smaller generalization gaps for image-text models such as
BASIC.

研究使用 BASIC 方法将对比学习框架在三个方面进行扩展：数据规模、模型规模和批次大小，并取得了对比学习模型的极高准确性和健壮性。