Background: We are witnessing an increasing adoption of machine learning
(ML), especially deep learning (DL) algorithms in many software systems,
including safety-critical systems such as health care systems or autonomous
driving vehicles. Ensuring the software quality of these systems is yet an open
challenge for the research community, mainly due to the inductive nature of ML
software systems. Traditionally, software systems were constructed deductively,
by writing down the rules that govern the behavior of the system as program
code. However, for ML software, these rules are inferred from training data.
Few recent research advances in the quality assurance of ML systems have
adapted different concepts from traditional software testing, such as mutation
testing, to help improve the reliability of ML software systems. However, it is
unclear if any of these proposed testing techniques from research are adopted
in practice. There is little empirical evidence about the testing strategies of
ML engineers. Aims: To fill this gap, we perform the first fine-grained
empirical study on ML testing practices in the wild, to identify the ML
properties being tested, the followed testing strategies, and their
implementation throughout the ML workflow. Method: First, we systematically
summarized the different testing strategies (e.g., Oracle Approximation), the
tested ML properties (e.g., Correctness, Bias, and Fairness), and the testing
methods (e.g., Unit test) from the literature. Then, we conducted a study to
understand the practices of testing ML software. Results: In our findings: 1)
we identified four (4) major categories of testing strategy including Grey-box,
White-box, Black-box, and Heuristic-based techniques that are used by the ML
engineers to find software bugs. 2) We identified 16 ML properties that are
tested in the ML workflow.

我们进行了第一项细致入微的实证研究，以填补现有研究中对机器学习软件测试实践的知识空白，从而识别机器学习质量属性、测试策略以及它们在机器学习工作流中的实施。

研究机器学习软件测试实践

Studying the Practices of Testing Machine Learning Software in the Wild

Having efficient testing strategies is a core challenge that needs to be
overcome for the release of automated driving. This necessitates clear
requirements as well as suitable methods for testing. In this work, the
requirements for perception modules are considered with respect to relevance.
The concept of relevance currently remains insufficiently defined and
specified. In this paper, we propose a novel methodology to overcome this
challenge by exemplary application to collision safety in the highway domain.
Using this general system and use case specification, a corresponding concept
for relevance is derived. Irrelevant objects are thus defined as objects which
do not limit the set of safe actions available to the ego vehicle under
consideration of all uncertainties. As an initial step, the use case is
decomposed into functional scenarios with respect to collision relevance. For
each functional scenario, possible actions of both the ego vehicle and any
other dynamic object are formalized as equations. This set of possible actions
is constrained by traffic rules, yielding relevance criteria. As a result, we
present a conservative estimation which dynamic objects are relevant for
perception and need to be considered for a complete evaluation. The estimation
provides requirements which are applicable for offline testing and validation
of perception components. A visualization is presented for examples from the
highD dataset, showing the plausibility of the results. Finally, a possibility
for a future validation of the presented relevance concept is outlined.

为了自动驾驶释放，需要有效的测试策略和适当的测试方法。本研究提出了一个新的方法来解决感知模块的要求问题，并针对碰撞安全在高速公路领域进行了示例应用。通过功能场景的拆分和限制交通规则，得出了感知的相关性标准。这些结果可应用于离线测试和验证。