Despite decades of research and practice in automated software testing, several fundamental concepts remain ill-defined and under-explored, yet offer enormous potential real-world impact. We show that these concepts raise exciting new challenges in the context of Large Language Models for software test generation. More specifically, we formally define and investigate the properties of hardening and catching tests. A hardening test is one that seeks to protect against future regressions, while a catching test is one that catches such a regression or a fault in new functionality introduced by a code change. Hardening tests can be generated at any time and may become catching tests when a future regression is caught. We also define and motivate the Catching `Just-in-Time' (JiTTest) Challenge, in which tests are generated `just-in-time' to catch new faults before they land into production. We show that any solution to Catching JiTTest generation can also be repurposed to catch latent faults in legacy code. We enumerate possible outcomes for hardening and catching tests and JiTTests, and discuss open research problems, deployment options, and initial results from our work on automated LLM-based hardening at Meta. This paper\footnote{Author order is alphabetical. The corresponding author is Mark Harman.} was written to accompany the keynote by the authors at the ACM International Conference on the Foundations of Software Engineering (FSE) 2025.

本研究解决了自动化软件测试中一些基本概念不明确且未被充分探讨的问题，尤其是在大型语言模型生成软件测试的背景下。论文正式定义了“硬化测试”和“捕获测试”的属性，并提出了即时捕获（JiTTest）挑战，展示了如何在生产环境之前生成测试以捕捉新故障。这项工作为未来软件测试的改进提供了新的思路和潜在的实用影响。

基于大型语言模型的软件测试的即时保证：开放研究挑战