Deep learning models have proven to be successful in a wide range of machine
learning tasks. Yet, they are often highly sensitive to perturbations on the
input data which can lead to incorrect decisions with high confidence,
hampering their deployment for practical use-cases. Thus, finding architectures
that are (more) robust against perturbations has received much attention in
recent years. Just like the search for well-performing architectures in terms
of clean accuracy, this usually involves a tedious trial-and-error process with
one additional challenge: the evaluation of a network's robustness is
significantly more expensive than its evaluation for clean accuracy. Thus, the
aim of this paper is to facilitate better streamlined research on architectural
design choices with respect to their impact on robustness as well as, for
example, the evaluation of surrogate measures for robustness. We therefore
borrow one of the most commonly considered search spaces for neural
architecture search for image classification, NAS-Bench-201, which contains a
manageable size of 6466 non-isomorphic network designs. We evaluate all these
networks on a range of common adversarial attacks and corruption types and
introduce a database on neural architecture design and robustness evaluations.
We further present three exemplary use cases of this dataset, in which we (i)
benchmark robustness measurements based on Jacobian and Hessian matrices for
their robustness predictability, (ii) perform neural architecture search on
robust accuracies, and (iii) provide an initial analysis of how architectural
design choices affect robustness. We find that carefully crafting the topology
of a network can have substantial impact on its robustness, where networks with
the same parameter count range in mean adversarial robust accuracy from
20%-41%. Code and data is available at this http URL

本文研究探讨了寻找更具有鲁棒性的体系结构对于减少深度学习模型在实际应用中对于输入数据微小扰动导致的错误决策的重要性。在评估鲁棒性方面，我们基于 NAS-Bench-201 进行神经网络架构搜索，对 6466 种不同的网络设计进行评估并引入数据库，发现对神经网络的拓扑结构进行优化可以显著提高其鲁棒性。