Deep learning (DL) defines a new data-driven programming paradigm where the
internal system logic is largely shaped by the training data. The standard way
of evaluating DL models is to examine their performance on a test dataset. The
quality of the test dataset is of great importance to gain confidence of the
trained models. Using an inadequate test dataset, DL models that have achieved
high test accuracy may still lack generality and robustness. In traditional
software testing, mutation testing is a well-established technique for quality
evaluation of test suites, which analyzes to what extent a test suite detects
the injected faults. However, due to the fundamental difference between
traditional software and deep learning-based software, traditional mutation
testing techniques cannot be directly applied to DL systems. In this paper, we
propose a mutation testing framework specialized for DL systems to measure the
quality of test data. To do this, by sharing the same spirit of mutation
testing in traditional software, we first define a set of source-level mutation
operators to inject faults to the source of DL (i.e., training data and
training programs). Then we design a set of model-level mutation operators that
directly inject faults into DL models without a training process. Eventually,
the quality of test data could be evaluated from the analysis on to what extent
the injected faults could be detected. The usefulness of the proposed mutation
testing techniques is demonstrated on two public datasets, namely MNIST and
CIFAR-10, with three DL models.

本研究提出了一种专为深度学习系统设计的突变测试框架，通过注入故障来度量测试数据的质量，并在 MNIST 和 CIFAR-10 数据集上验证了其有效性。