The field of Deep Learning is rich with empirical evidence of human-like
performance on a variety of prediction tasks. However, despite these successes,
the recent Predicting Generalization in Deep Learning (PGDL) NeurIPS 2020
competition suggests that there is a need for more robust and efficient
measures of network generalization. In this work, we propose a new framework
for evaluating the generalization capabilities of trained networks. We use
perturbation response (PR) curves that capture the accuracy change of a given
network as a function of varying levels of training sample perturbation. From
these PR curves, we derive novel statistics that capture generalization
capability. Specifically, we introduce two new measures for accurately
predicting generalization gaps: the Gi-score and Pal-score, which are inspired
by the Gini coefficient and Palma ratio (measures of income inequality), that
accurately predict generalization gaps. Using our framework applied to intra
and inter-class sample mixup, we attain better predictive scores than the
current state-of-the-art measures on a majority of tasks in the PGDL
competition. In addition, we show that our framework and the proposed
statistics can be used to capture to what extent a trained network is invariant
to a given parametric input transformation, such as rotation or translation.
Therefore, these generalization gap prediction statistics also provide a useful
means for selecting optimal network architectures and hyperparameters that are
invariant to a certain perturbation.

本文提出了一种用于评估神经网络泛化能力的新框架，使用扰动响应曲线和 Gi-score，Pal-score 等新指标来预测泛化差距，并在 PGDL 竞赛中取得更好的预测成绩。此外，该框架和指标还可用于选择与给定扰动不变的最佳网络架构和超参数。

使用扰动响应曲线预测深度神经网络的泛化能力

Predicting Deep Neural Network Generalization with Perturbation Response  Curves

The field of Deep Learning is rich with empirical evidence of human-like
performance on a variety of regression, classification, and control tasks.
However, despite these successes, the field lacks strong theoretical error
bounds and consistent measures of network generalization and learned
invariances. In this work, we introduce two new measures, the Gi-score and
Pal-score, that capture a deep neural network's generalization capabilities.
Inspired by the Gini coefficient and Palma ratio, measures of income
inequality, our statistics are robust measures of a network's invariance to
perturbations that accurately predict generalization gaps, i.e., the difference
between accuracy on training and test sets.

本文提出了 Gi-score 和 Pal-score 两个新的度量标准，用于衡量深度神经网络的泛化能力和不变性，这些度量标准可准确预测一些泛化差距。