There remains a list of unanswered research questions on deep learning (DL),
including the remarkable generalization power of overparametrized neural
networks, the efficient optimization performance despite the non-convexity, and
the mechanisms behind flat minima in generalization. In this paper, we adopt an
information-theoretic perspective to explore the theoretical foundations of
supervised classification using deep neural networks (DNNs). Our analysis
introduces the concepts of fitting error and model risk, which, together with
generalization error, constitute an upper bound on the expected risk. We
demonstrate that the generalization errors are bounded by the complexity,
influenced by both the smoothness of distribution and the sample size.
Consequently, task complexity serves as a reliable indicator of the dataset's
quality, guiding the setting of regularization hyperparameters. Furthermore,
the derived upper bound fitting error links the back-propagated gradient,
Neural Tangent Kernel (NTK), and the model's parameter count with the fitting
error. Utilizing the triangle inequality, we establish an upper bound on the
expected risk. This bound offers valuable insights into the effects of
overparameterization, non-convex optimization, and the flat minima in
DNNs.Finally, empirical verification confirms a significant positive
correlation between the derived theoretical bounds and the practical expected
risk, confirming the practical relevance of the theoretical findings.

采用信息理论的视角探索深度神经网络在有监督分类中的理论基础，分析了拟合误差、模型风险和泛化误差上界的相关概念及其对样本数据质量和正则化超参数设置的指导作用，研究发现过度参数化、非凸优化和平坦极小值在深度神经网络中的影响，并通过实证验证证实了理论发现与实际风险之间显著的正相关关系。

监督分类的信息论视角下的误差边界

Error Bounds of Supervised Classification from Information-Theoretic  Perspective

Data augmentation has been widely applied as an effective methodology to
improve generalization in particular when training deep neural networks.
Recently, researchers proposed a few intensive data augmentation techniques,
which indeed improved accuracy, yet we notice that these methods augment data
have also caused a considerable gap between clean and augmented data. In this
paper, we revisit this problem from an analytical perspective, for which we
estimate the upper-bound of expected risk using two terms, namely, empirical
risk and generalization error, respectively. We develop an understanding of
data augmentation as regularization, which highlights the major features. As a
result, data augmentation significantly reduces the generalization error, but
meanwhile leads to a slightly higher empirical risk. On the assumption that
data augmentation helps models converge to a better region, the model can
benefit from a lower empirical risk achieved by a simple method, i.e., using
less-augmented data to refine the model trained on fully-augmented data. Our
approach achieves consistent accuracy gain on a few standard image
classification benchmarks, and the gain transfers to object detection.

利用数据增强作为正则化方式，我们从解析角度重新考虑了数据增强在深度神经网络中的泛化问题，并发现它显著降低了模型泛化误差，但同时也导致略微增加了经验风险。使用较少的数据对完整增强的数据进行训练能够帮助模型更好地收敛于更好的区域，从而进一步提高准确性。