This paper considers "model diagnosis", which we formulate as a
classification problem. Given a pre-trained neural network (NN), the goal is to
predict the source of failure from a set of failure modes (such as a wrong
hyperparameter, inadequate model size, and insufficient data) without knowing
the training configuration of the pre-trained NN. The conventional diagnosis
approach uses training and validation errors to determine whether the model is
underfitting or overfitting. However, we show that rich information about NN
performance is encoded in the optimization loss landscape, which provides more
actionable insights than validation-based measurements. Therefore, we propose a
diagnosis method called MD tree based on loss landscape metrics and
experimentally demonstrate its advantage over classical validation-based
approaches. We verify the effectiveness of MD tree in multiple practical
scenarios: (1) use several models trained on one dataset to diagnose a model
trained on another dataset, essentially a few-shot dataset transfer problem;
(2) use small models (or models trained with small data) to diagnose big models
(or models trained with big data), essentially a scale transfer problem. In a
dataset transfer task, MD tree achieves an accuracy of 87.7%, outperforming
validation-based approaches by 14.88%. Our code is available at
this https URL

该论文提出了一种基于损失函数空间度量的模型诊断方法，通过预测故障源，解决了模型训练实验配置未知的情况下的分类问题，并在实践中验证了其优势。

MD 树：基于损失景观生长的模型诊断树

MD tree: a model-diagnostic tree grown on loss landscape

Recent works have shown considerable improvements in task-oriented dialogue
(TOD) systems by utilizing pretrained large language models (LLMs) in an
end-to-end manner. However, the biased behavior of each component in a TOD
system and the error propagation issue in the end-to-end framework can lead to
seriously biased TOD responses. Existing works of fairness only focus on the
total bias of a system. In this paper, we propose a diagnosis method to
attribute bias to each component of a TOD system. With the proposed attribution
method, we can gain a deeper understanding of the sources of bias.
Additionally, researchers can mitigate biased model behavior at a more granular
level. We conduct experiments to attribute the TOD system's bias toward three
demographic axes: gender, age, and race. Experimental results show that the
bias of a TOD system usually comes from the response generation model.

使用预训练的大型语言模型，本文提出了一种诊断方法来确定任务导向对话系统中每个组件的偏见，并通过实验结果表明这种偏见主要来自回应生成模型。