Recent developments in neural architecture search (NAS) emphasize the
significance of considering robust architectures against malicious data.
However, there is a notable absence of benchmark evaluations and theoretical
guarantees for searching these robust architectures, especially when
adversarial training is considered. In this work, we aim to address these two
challenges, making twofold contributions. First, we release a comprehensive
data set that encompasses both clean accuracy and robust accuracy for a vast
array of adversarially trained networks from the NAS-Bench-201 search space on
image datasets. Then, leveraging the neural tangent kernel (NTK) tool from deep
learning theory, we establish a generalization theory for searching
architecture in terms of clean accuracy and robust accuracy under
multi-objective adversarial training. We firmly believe that our benchmark and
theoretical insights will significantly benefit the NAS community through
reliable reproducibility, efficient assessment, and theoretical foundation,
particularly in the pursuit of robust architectures.

最近神经网络架构搜索 (NAS) 的发展强调考虑恶意数据下的鲁棒性架构的重要性，然而，在搜索这些鲁棒性架构时，尤其是在考虑对抗训练时，缺乏显著的基准评估和理论保证。本文旨在解决这两个挑战，做出了双重贡献：首先，我们发布了一个综合数据集，该数据集包含来自 NAS-Bench-201 搜索空间在图像数据集上经过对抗训练的网络的干净准确度和鲁棒准确度；其次，利用深度学习理论中的神经切线核 (NTK) 工具，我们建立了一个关于在干净准确度和鲁棒准确度下搜索架构的泛化理论，该理论适用于多目标对抗训练。我们坚信，我们的基准和理论见解将对 NAS 社区产生重大影响，通过可靠的再现性、高效的评估和理论基础，特别是在追求鲁棒架构方面。

鲁棒性对抗训练下的 NAS：基准、理论与进展

Robust NAS under adversarial training: benchmark, theory, and beyond

The measurement of progress using benchmarks evaluations is ubiquitous in
computer science and machine learning. However, common approaches to analyzing
and presenting the results of benchmark comparisons of multiple algorithms over
multiple datasets, such as the critical difference diagram introduced by
Dem\v{s}ar (2006), have important shortcomings and, we show, are open to both
inadvertent and intentional manipulation. To address these issues, we propose a
new approach to presenting the results of benchmark comparisons, the Multiple
Comparison Matrix (MCM), that prioritizes pairwise comparisons and precludes
the means of manipulating experimental results in existing approaches. MCM can
be used to show the results of an all-pairs comparison, or to show the results
of a comparison between one or more selected algorithms and the state of the
art. MCM is implemented in Python and is publicly available.

本文章介绍了一个新的基于配对比较的方法来呈现基准测试的结果，并排除了现有方法中对实验结果进行意外或有意动态的可能性。