Designing robust machine learning systems remains an open problem, and there
is a need for benchmark problems that cover both environmental changes and
evaluation on a downstream task. In this work, we introduce AVOIDDS, a
realistic object detection benchmark for the vision-based aircraft
detect-and-avoid problem. We provide a labeled dataset consisting of 72,000
photorealistic images of intruder aircraft with various lighting conditions,
weather conditions, relative geometries, and geographic locations. We also
provide an interface that evaluates trained models on slices of this dataset to
identify changes in performance with respect to changing environmental
conditions. Finally, we implement a fully-integrated, closed-loop simulator of
the vision-based detect-and-avoid problem to evaluate trained models with
respect to the downstream collision avoidance task. This benchmark will enable
further research in the design of robust machine learning systems for use in
safety-critical applications. The AVOIDDS dataset and code are publicly
available at
$\href{this https URL}{purl.stanford.edu/hj293cv5980}$
and
$\href{this https URL}{github.com/sisl/VisionBasedAircraftDAA}$,
respectively.

本文介绍了 AVOIDDS，这是一个逼真的目标检测基准测试，包含了 72000 个逼真的图片数据集，以及相应的模型评估接口和整个仿真问题的解决方案，在安全关键应用中，这样的基准测试将有助于设计稳健的机器学习系统。