This paper considers "model diagnosis", which we formulate as a
classification problem. Given a pre-trained neural network (NN), the goal is to
predict the source of failure from a set of failure modes (such as a wrong
hyperparameter, inadequate model size, and insufficient data) without knowing
the training configuration of the pre-trained NN. The conventional diagnosis
approach uses training and validation errors to determine whether the model is
underfitting or overfitting. However, we show that rich information about NN
performance is encoded in the optimization loss landscape, which provides more
actionable insights than validation-based measurements. Therefore, we propose a
diagnosis method called MD tree based on loss landscape metrics and
experimentally demonstrate its advantage over classical validation-based
approaches. We verify the effectiveness of MD tree in multiple practical
scenarios: (1) use several models trained on one dataset to diagnose a model
trained on another dataset, essentially a few-shot dataset transfer problem;
(2) use small models (or models trained with small data) to diagnose big models
(or models trained with big data), essentially a scale transfer problem. In a
dataset transfer task, MD tree achieves an accuracy of 87.7%, outperforming
validation-based approaches by 14.88%. Our code is available at
this https URL

该论文提出了一种基于损失函数空间度量的模型诊断方法，通过预测故障源，解决了模型训练实验配置未知的情况下的分类问题，并在实践中验证了其优势。

MD 树：基于损失景观生长的模型诊断树

MD tree: a model-diagnostic tree grown on loss landscape

In practice, metric analysis on a specific train and test dataset does not
guarantee reliable or fair ML models. This is partially due to the fact that
obtaining a balanced, diverse, and perfectly labeled dataset is typically
expensive, time-consuming, and error-prone. Rather than relying on a carefully
designed test set to assess ML models' failures, fairness, or robustness, this
paper proposes Semantic Image Attack (SIA), a method based on the adversarial
attack that provides semantic adversarial images to allow model diagnosis,
interpretability, and robustness. Traditional adversarial training is a popular
methodology for robustifying ML models against attacks. However, existing
adversarial methods do not combine the two aspects that enable the
interpretation and analysis of the model's flaws: semantic traceability and
perceptual quality. SIA combines the two features via iterative gradient ascent
on a predefined semantic attribute space and the image space. We illustrate the
validity of our approach in three scenarios for keypoint detection and
classification. (1) Model diagnosis: SIA generates a histogram of attributes
that highlights the semantic vulnerability of the ML model (i.e., attributes
that make the model fail). (2) Stronger attacks: SIA generates adversarial
examples with visually interpretable attributes that lead to higher attack
success rates than baseline methods. The adversarial training on SIA improves
the transferable robustness across different gradient-based attacks. (3)
Robustness to imbalanced datasets: we use SIA to augment the underrepresented
classes, which outperforms strong augmentation and re-balancing baselines.

该论文提出了一种基于语义对抗攻击的方法，即 SIA，它通过在预定义的语义属性空间和图像空间上的迭代梯度上升来结合语义追踪和感知质量这两个能够解释和分析模型缺陷的特征。该方法在模型诊断、攻击成功率以及不平衡数据鲁棒性方面具有很好的应用前景。