Evaluating the performance of machine learning models on diverse and
underrepresented subgroups is essential for ensuring fairness and reliability
in real-world applications. However, accurately assessing model performance
becomes challenging due to two main issues: (1) a scarcity of test data,
especially for small subgroups, and (2) possible distributional shifts in the
model's deployment setting, which may not align with the available test data.
In this work, we introduce 3S Testing, a deep generative modeling framework to
facilitate model evaluation by generating synthetic test sets for small
subgroups and simulating distributional shifts. Our experiments demonstrate
that 3S Testing outperforms traditional baselines -- including real test data
alone -- in estimating model performance on minority subgroups and under
plausible distributional shifts. In addition, 3S offers intervals around its
performance estimates, exhibiting superior coverage of the ground truth
compared to existing approaches. Overall, these results raise the question of
whether we need a paradigm shift away from limited real test data towards
synthetic test data.

评估机器学习模型在多样化和少数群体上的性能对于确保公平和可靠的实际应用至关重要。本研究引入了一种深度生成建模框架 3S Testing，通过为小型子群生成合成测试集并模拟分布偏移，促进模型评估。实验证明，3S Testing 在估计少数群体和可能的分布偏移的模型性能方面优于传统基准，包括仅使用真实测试数据，而且 3S 的性能估计还提供了区间，相对于现有方法，具有更好的准确性。总的来说，这些结果引发了一个问题，是否需要从有限的真实测试数据转向合成测试数据的新方法。