The predominant de facto paradigm of testing ML models relies on either using only held-out data to compute aggregate evaluation metrics or by assessing the performance on different subgroups. However, such data-only testing methods operate under the restrictive assumption that the available empirical data is the sole input for testing ML models, disregarding valuable contextual information that could guide model testing. In this paper, we challenge the go-to approach of data-only testing and introduce context-aware testing (CAT) which uses context as an inductive bias to guide the search for meaningful model failures. We instantiate the first CAT system, SMART Testing, which employs large language models to hypothesize relevant and likely failures, which are evaluated on data using a self-falsification mechanism. Through empirical evaluations in diverse settings, we show that SMART automatically identifies more relevant and impactful failures than alternatives, demonstrating the potential of CAT as a testing paradigm.

本研究针对现有的模型测试方法仅依赖保留数据进行评估的限制，提出了基于上下文的测试（CAT）方法，该方法利用上下文信息指导模型失败的搜索。通过构建SMART测试系统，利用大型语言模型识别相关和可能的失败，实验证明CAT在识别失败方面的有效性，展现了其作为测试新范式的潜力。

基于上下文的测试：大型语言模型的模型测试新范式