Much research on Machine Learning testing relies on empirical studies that
evaluate and show their potential. However, in this context empirical results
are sensitive to a number of parameters that can adversely impact the results
of the experiments and potentially lead to wrong conclusions (Type I errors,
i.e., incorrectly rejecting the Null Hypothesis). To this end, we survey the
related literature and identify 10 commonly adopted empirical evaluation
hazards that may significantly impact experimental results. We then perform a
sensitivity analysis on 30 influential studies that were published in top-tier
SE venues, against our hazard set and demonstrate their criticality. Our
findings indicate that all 10 hazards we identify have the potential to
invalidate experimental findings, such as those made by the related literature,
and should be handled properly. Going a step further, we propose a point set of
10 good empirical practices that has the potential to mitigate the impact of
the hazards. We believe our work forms the first step towards raising awareness
of the common pitfalls and good practices within the software engineering
community and hopefully contribute towards setting particular expectations for
empirical research in the field of deep learning testing.

这篇论文调查了机器学习测试中常见的实证评估隐患，通过敏感性分析展示了这些隐患对实验结果的重要性，并提出了一套良好的实证实践以减轻这些隐患的影响。这项研究对软件工程社区的共同问题和良好实践进行了初步调查，并在深度学习测试领域为实证研究设定了特定的期望。

深度学习测试中的风险：普遍性、影响和建议

Hazards in Deep Learning Testing: Prevalence, Impact and Recommendations

There is a growing interest in industry and academia in machine learning (ML)
testing. We believe that industry and academia need to learn together to
produce rigorous and relevant knowledge. In this study, we initiate a
collaboration between stakeholders from one case company, one research
institute, and one university. To establish a common view of the problem
domain, we applied an interactive rapid review of the state of the art. Four
researchers from Lund University and RISE Research Institutes and four
practitioners from Axis Communications reviewed a set of 180 primary studies on
ML testing. We developed a taxonomy for the communication around ML testing
challenges and results and identified a list of 12 review questions relevant
for Axis Communications. The three most important questions (data testing,
metrics for assessment, and test generation) were mapped to the literature, and
an in-depth analysis of the 35 primary studies matching the most important
question (data testing) was made. A final set of the five best matches were
analysed and we reflect on the criteria for applicability and relevance for the
industry. The taxonomies are helpful for communication but not final.
Furthermore, there was no perfect match to the case company's investigated
review question (data testing). However, we extracted relevant approaches from
the five studies on a conceptual level to support later context-specific
improvements. We found the interactive rapid review approach useful for
triggering and aligning communication between the different stakeholders.

本研究旨在探究机器学习测试的相关问题，并通过快速审核等方法建立了与业界、学术界等相关方的共同视角，分析了一系列文献，建立了相关分类学，并提出了 12 个问题用于检验与测试机器学习的质量及可行性，针对数据测试等方面进行了深入研究，并从五个研究中提取相关方法以支持日后的改进工作。

实践中的机器学习测试 -- 从与 Axis Communications 的互动快速审查中学到的经验教训

Exploring ML testing in practice -- Lessons learned from an interactive rapid review with Axis Communications

This paper provides a comprehensive survey of Machine Learning Testing (ML
testing) research. It covers 144 papers on testing properties (e.g.,
correctness, robustness, and fairness), testing components (e.g., the data,
learning program, and framework), testing workflow (e.g., test generation and
test evaluation), and application scenarios (e.g., autonomous driving, machine
translation). The paper also analyses trends concerning datasets, research
trends, and research focus, concluding with research challenges and promising
research directions in ML testing.

本文综述了 144 篇关于机器学习测试（ML Testing）的研究论文，涵盖了测试属性（如正确性，鲁棒性和公平性）、测试组件（如数据、学习程序和框架）、测试工作流（如测试生成和测试评估）和应用场景（如自动驾驶、机器翻译）。研究还分析了数据集、研究趋势和研究重点的趋势，并得出了 ML 测试中的研究挑战和有前景的研究方向。