Deep Neural Networks (DNN) are core components for classification and
regression tasks of many software systems. Companies incur in high costs for
testing DNN with datasets representative of the inputs expected in operation,
as these need to be manually labelled. The challenge is to select a
representative set of test inputs as small as possible to reduce the labelling
cost, while sufficing to yield unbiased high-confidence estimates of the
expected DNN accuracy. At the same time, testers are interested in exposing as
many DNN mispredictions as possible to improve the DNN, ending up in the need
for techniques pursuing a threefold aim: small dataset size, trustworthy
estimates, mispredictions exposure. This study presents DeepSample, a family of
DNN testing techniques for cost-effective accuracy assessment based on
probabilistic sampling. We investigate whether, to what extent, and under which
conditions probabilistic sampling can help to tackle the outlined challenge. We
implement five new sampling-based testing techniques, and perform a
comprehensive comparison of such techniques and of three further
state-of-the-art techniques for both DNN classification and regression tasks.
Results serve as guidance for best use of sampling-based testing for faithful
and high-confidence estimates of DNN accuracy in operation at low cost.

通过概率抽样，研究了五种新的基于抽样的测试技术以及其他三种先进技术，以实现对深度神经网络在运行中准确度的忠实和高置信度估计，从而降低成本。

DeepSample：基于 DNN 的操作准确性评估测试

DeepSample: DNN sampling-based testing for operational accuracy  assessment

Despite their unprecedented success, DNNs are notoriously fragile to small
shifts in data distribution, demanding effective testing techniques that can
assess their dependability. Despite recent advances in DNN testing, there is a
lack of systematic testing approaches that assess the DNN's capability to
generalise and operate comparably beyond data in their training distribution.
We address this gap with DeepKnowledge, a systematic testing methodology for
DNN-based systems founded on the theory of knowledge generalisation, which aims
to enhance DNN robustness and reduce the residual risk of 'black box' models.
Conforming to this theory, DeepKnowledge posits that core computational DNN
units, termed Transfer Knowledge neurons, can generalise under domain shift.
DeepKnowledge provides an objective confidence measurement on testing
activities of DNN given data distribution shifts and uses this information to
instrument a generalisation-informed test adequacy criterion to check the
transfer knowledge capacity of a test set. Our empirical evaluation of several
DNNs, across multiple datasets and state-of-the-art adversarial generation
techniques demonstrates the usefulness and effectiveness of DeepKnowledge and
its ability to support the engineering of more dependable DNNs. We report
improvements of up to 10 percentage points over state-of-the-art coverage
criteria for detecting adversarial attacks on several benchmarks, including
MNIST, SVHN, and CIFAR.

通过 DeepKnowledge 系统测试方法，可以评估 DNN 的可靠性和泛化能力，并通过提供测量信息和检查测试集的传输知识容量来改善 DNN 的稳健性。

深度知识：基于泛化的深度学习测试

DeepKnowledge: Generalisation-Driven Deep Learning Testing

Deep learning (DL) has achieved remarkable progress over the past decade and
been widely applied to many safety-critical applications. However, the
robustness of DL systems recently receives great concerns, such as adversarial
examples against computer vision systems, which could potentially result in
severe consequences. Adopting testing techniques could help to evaluate the
robustness of a DL system and therefore detect vulnerabilities at an early
stage. The main challenge of testing such systems is that its runtime state
space is too large: if we view each neuron as a runtime state for DL, then a DL
system often contains massive states, rendering testing each state almost
impossible. For traditional software, combinatorial testing (CT) is an
effective testing technique to reduce the testing space while obtaining
relatively high defect detection abilities. In this paper, we perform an
exploratory study of CT on DL systems. We adapt the concept in CT and propose a
set of coverage criteria for DL systems, as well as a CT coverage guided test
generation technique. Our evaluation demonstrates that CT provides a promising
avenue for testing DL systems. We further pose several open questions and
interesting directions for combinatorial testing of DL systems.

本文探讨了如何采用组合测试技术来测试深度学习系统，以提高其鲁棒性，从而及早发现漏洞。作者提出了一套 DL 系统覆盖度准则和 CT 覆盖度引导测试生成技术，并通过实验证明组合测试可以有效地降低测试空间并提高其缺陷检测能力。此外，还对深度学习系统的组合测试提出了一些未解之问和有趣的方向。

深度学习系统的组合测试

Combinatorial Testing for Deep Learning Systems

While Deep Neural Networks (DNNs) have established the fundamentals of
DNN-based autonomous driving systems, they may exhibit erroneous behaviors and
cause fatal accidents. To resolve the safety issues of autonomous driving
systems, a recent set of testing techniques have been designed to automatically
generate test cases, e.g., new input images transformed from the original ones.
Unfortunately, many such generated input images often render inferior
authenticity, lacking accurate semantic information of the driving scenes and
hence compromising the resulting efficacy and reliability.
In this paper, we propose DeepRoad, an unsupervised framework to
automatically generate large amounts of accurate driving scenes to test the
consistency of DNN-based autonomous driving systems across different scenes. In
particular, DeepRoad delivers driving scenes with various weather conditions
(including those with rather extreme conditions) by applying the Generative
Adversarial Networks (GANs) along with the corresponding real-world weather
scenes. Moreover, we have implemented DeepRoad to test three well-recognized
DNN-based autonomous driving systems. Experimental results demonstrate that
DeepRoad can detect thousands of behavioral inconsistencies in these systems.

本文提出了 DeepRoad，是一个无监督的框架，它可以通过使用生成对抗网络以及对应的现实天气场景，自动生成各种天气条件下的准确驾驶场景，用于测试 DNN 自动驾驶系统的一致性，并在三个认可的 DNN 自动驾驶系统上进行了实验，证明了 DeepRoad 可以检测出数千种行为不一致性。