Practical object detection application can lose its effectiveness on image
inputs with natural distribution shifts. This problem leads the research
community to pay more attention on the robustness of detectors under
Out-Of-Distribution (OOD) inputs. Existing works construct datasets to
benchmark the detector's OOD robustness for a specific application scenario,
e.g., Autonomous Driving. However, these datasets lack universality and are
hard to benchmark general detectors built on common tasks such as COCO. To give
a more comprehensive robustness assessment, we introduce
COCO-O(ut-of-distribution), a test dataset based on COCO with 6 types of
natural distribution shifts. COCO-O has a large distribution gap with training
data and results in a significant 55.7% relative performance drop on a Faster
R-CNN detector. We leverage COCO-O to conduct experiments on more than 100
modern object detectors to investigate if their improvements are credible or
just over-fitting to the COCO test set. Unfortunately, most classic detectors
in early years do not exhibit strong OOD generalization. We further study the
robustness effect on recent breakthroughs of detector's architecture design,
augmentation and pre-training techniques. Some empirical findings are revealed:
1) Compared with detection head or neck, backbone is the most important part
for robustness; 2) An end-to-end detection transformer design brings no
enhancement, and may even reduce robustness; 3) Large-scale foundation models
have made a great leap on robust object detection. We hope our COCO-O could
provide a rich testbed for robustness study of object detection. The dataset
will be available at
https://github.com/alibaba/easyrobust/tree/main/benchmarks/coco_o.

通过引入基于 COCO 的 COCO-O 测试数据集，研究了 100 多种现代目标检测器的鲁棒性效果，并发现了各种架构设计、数据增强和预训练技术对鲁棒性的影响。