Modern deep neural networks tend to be evaluated on static test sets. One
shortcoming of this is the fact that these deep neural networks cannot be
easily evaluated for robustness issues with respect to specific scene
variations. For example, it is hard to study the robustness of these networks
to variations of object scale, object pose, scene lighting and 3D occlusions.
The main reason is that collecting real datasets with fine-grained naturalistic
variations of sufficient scale can be extremely time-consuming and expensive.
In this work, we present Counterfactual Simulation Testing, a counterfactual
framework that allows us to study the robustness of neural networks with
respect to some of these naturalistic variations by building realistic
synthetic scenes that allow us to ask counterfactual questions to the models,
ultimately providing answers to questions such as "Would your classification
still be correct if the object were viewed from the top?" or "Would your
classification still be correct if the object were partially occluded by
another object?". Our method allows for a fair comparison of the robustness of
recently released, state-of-the-art Convolutional Neural Networks and Vision
Transformers, with respect to these naturalistic variations. We find evidence
that ConvNext is more robust to pose and scale variations than Swin, that
ConvNext generalizes better to our simulated domain and that Swin handles
partial occlusion better than ConvNext. We also find that robustness for all
networks improves with network scale and with data scale and variety. We
release the Naturalistic Variation Object Dataset (NVD), a large simulated
dataset of 272k images of everyday objects with naturalistic variations such as
object pose, scale, viewpoint, lighting and occlusions. Project page:
this https URL

本研究提出了反事实模拟测试的框架，并使用构建的逼真合成场景研究深度神经网络对特定情景变化的鲁棒性问题，发现随着网络规模和数据规模多样性的增加，所有网络的鲁棒性均有所提高，并通过对比发现 ConvNext 对姿态和尺度变化更加鲁棒，而 Swin 对部分遮挡更加鲁棒，同时发布了包含 272k 张自然变形的对象图片的数据集（NVD）