Simulators are widely used to test Autonomous Driving Systems (ADS), but
their potential flakiness can lead to inconsistent test results. We investigate
test flakiness in simulation-based testing of ADS by addressing two key
questions: (1) How do flaky ADS simulations impact automated testing that
relies on randomized algorithms? and (2) Can machine learning (ML) effectively
identify flaky ADS tests while decreasing the required number of test reruns?
Our empirical results, obtained from two widely-used open-source ADS simulators
and five diverse ADS test setups, show that test flakiness in ADS is a common
occurrence and can significantly impact the test results obtained by randomized
algorithms. Further, our ML classifiers effectively identify flaky ADS tests
using only a single test run, achieving F1-scores of $85$%, $82$% and $96$% for
three different ADS test setups. Our classifiers significantly outperform our
non-ML baseline, which requires executing tests at least twice, by $31$%,
$21$%, and $13$% in F1-score performance, respectively. We conclude with a
discussion on the scope, implications and limitations of our study. We provide
our complete replication package in a Github repository.

自动驾驶系统模拟器的不稳定性会导致测试结果不一致，本研究通过研究两个关键问题：（1）不稳定模拟对基于随机算法的自动化测试的影响如何？（2）机器学习是否能够有效识别不稳定的测试并减少测试重新运行的次数？实证结果表明：自动驾驶系统测试的不稳定性普遍存在，并且能够显著影响基于随机算法的测试结果。此外，我们的机器学习分类器在只进行一次测试时能够有效识别不稳定的测试，在三个不同的自动驾驶系统测试中分别达到了 85％，82％和 96％的 F1 分数。我们的分类器在 F1 分数表现方面明显优于非机器学习基准模型，性能分别提高了 31％，21％和 13％。最后，我们对研究的范围、意义和局限性进行了讨论，并提供了完整的代码复制包。