Existing methods for isolating hard subpopulations and spurious correlations
in datasets often require human intervention. This can make these methods
labor-intensive and dataset-specific. To address these shortcomings, we present
a scalable method for automatically distilling a model's failure modes.
Specifically, we harness linear classifiers to identify consistent error
patterns, and, in turn, induce a natural representation of these failure modes
as directions within the feature space. We demonstrate that this framework
allows us to discover and automatically caption challenging subpopulations
within the training dataset. Moreover, by combining our framework with
off-the-shelf diffusion models, we can generate images that are especially
challenging for the analyzed model, and thus can be used to perform synthetic
data augmentation that helps remedy the model's failure modes. Code available
at this https URL

本文介绍了一种自动提取模型误差模式的方法：使用线性分类器来识别一致的错误模式，并将其表示为特征空间内的方向。使用此框架，结合扩散模型，能够自动发现训练数据集中具有挑战性的子人群，进而进行数据扩充，以改善模型失效的情况。