In many applications, one works with neural network models trained by someone
else. For such pretrained models, one may not have access to training data or
test data. Moreover, one may not know details about the model, e.g., the
specifics of the training data, the loss function, the hyperparameter values,
etc. Given one or many pretrained models, it is a challenge to say anything
about the expected performance or quality of the models. Here, we address this
challenge by providing a detailed meta-analysis of hundreds of
publicly-available pretrained models. We examine norm based capacity control
metrics as well as power law based metrics from the recently-developed Theory
of Heavy-Tailed Self Regularization. We find that norm based metrics correlate
well with reported test accuracies for well-trained models, but that they often
cannot distinguish well-trained versus poorly-trained models. We also find that
power law based metrics can do much better -- quantitatively better at
discriminating among series of well-trained models with a given architecture;
and qualitatively better at discriminating well-trained versus poorly-trained
models. These methods can be used to identify when a pretrained neural network
has problems that cannot be detected simply by examining training/test
accuracies.

该研究利用基于规范的容量控制指标和基于幂律的重尾自正则化理论的度量方法对数百个公开可用的预训练模型进行了详细的元分析，发现基于幂律的度量方法可以更好地区分预训练模型的良好培训和质量低劣的模型。