Deep neural networks (DNN) are increasingly applied in safety-critical
systems, e.g., for face recognition, autonomous car control and malware
detection. It is also shown that DNNs are subject to attacks such as
adversarial perturbation and thus must be properly tested. Many coverage
criteria for DNN since have been proposed, inspired by the success of code
coverage criteria for software programs. The expectation is that if a DNN is a
well tested (and retrained) according to such coverage criteria, it is more
likely to be robust. In this work, we conduct an empirical study to evaluate
the relationship between coverage, robustness and attack/defense metrics for
DNN. Our study is the largest to date and systematically done based on 100 DNN
models and 25 metrics. One of our findings is that there is limited correlation
between coverage and robustness, i.e., improving coverage does not help improve
the robustness. Our dataset and implementation have been made available to
serve as a benchmark for future studies on testing DNN.

通过实证研究 100 个 DNN 模型和 25 种指标，发现 DNN 的测试覆盖率与鲁棒性之间的相关性有限，即提高测试覆盖率并不能帮助提高鲁棒性。这个研究提出的数据集和实现也可以作为测试 DNN 的基准。

深度神经网络的覆盖率与鲁棒性之间存在有限的相关性

There is Limited Correlation between Coverage and Robustness for Deep  Neural Networks

Concolic testing combines program execution and symbolic analysis to explore
the execution paths of a software program. This paper presents the first
concolic testing approach for Deep Neural Networks (DNNs). More specifically,
we formalise coverage criteria for DNNs that have been studied in the
literature, and then develop a coherent method for performing concolic testing
to increase test coverage. Our experimental results show the effectiveness of
the concolic testing approach in both achieving high coverage and finding
adversarial examples.

本文提出了第一个针对深度神经网络（DNN）的连接符合测试方法，旨在提高测试覆盖率，并在实验证明了该方法在实现高覆盖率和发现对抗性示例方面的有效性。