This paper aims to theoretically analyze the complexity of feature
transformations encoded in piecewise linear DNNs with ReLU layers. We propose
metrics to measure three types of complexities of transformations based on the
information theory. We further discover and prove the strong correlation
between the complexity and the disentanglement of transformations. Based on the
proposed metrics, we analyze two typical phenomena of the change of the
transformation complexity during the training process, and explore the ceiling
of a DNN's complexity. The proposed metrics can also be used as a loss to learn
a DNN with the minimum complexity, which also controls the over-fitting level
of the DNN and influences adversarial robustness, adversarial transferability,
and knowledge consistency. Comprehensive comparative studies have provided new
perspectives to understand the DNN.

本文提出了一种使用信息论度量转换复杂度的指标，并探究了转换复杂度与 disentanglement 之间的强相关性，进一步分析了 DNN 训练过程中的两个典型现象，还探讨了 DNN 复杂度的上限，并使用提出的指标作为损失，学习具有最小复杂度的 DNN，并控制过拟合水平等，从而影响对抗鲁棒性、对抗传递性和知识一致性，同时提供了新的理解 DNN 的视角。

针对 ReLU DNN 的变换复杂度的理论分析

Towards Theoretical Analysis of Transformation Complexity of ReLU DNNs

We prove sharp dimension-free representation results for neural networks with
$D$ ReLU layers under square loss for a class of functions $\mathcal{G}_D$
defined in the paper. These results capture the precise benefits of depth in
the following sense:
1. The rates for representing the class of functions $\mathcal{G}_D$ via $D$
ReLU layers is sharp up to constants, as shown by matching lower bounds.
2. For each $D$, $\mathcal{G}_{D} \subseteq \mathcal{G}_{D+1}$ and as $D$
grows the class of functions $\mathcal{G}_{D}$ contains progressively less
smooth functions.
3. If $D^{\prime} < D$, then the approximation rate for the class
$\mathcal{G}_D$ achieved by depth $D^{\prime}$ networks is strictly worse than
that achieved by depth $D$ networks.
This constitutes a fine-grained characterization of the representation power
of feedforward networks of arbitrary depth $D$ and number of neurons $N$, in
contrast to existing representation results which either require $D$ growing
quickly with $N$ or assume that the function being represented is highly
smooth. In the latter case similar rates can be obtained with a single
nonlinear layer. Our results confirm the prevailing hypothesis that deeper
networks are better at representing less smooth functions, and indeed, the main
technical novelty is to fully exploit the fact that deep networks can produce
highly oscillatory functions with few activation functions.

该论文证明了具有 $D$ ReLU 层的神经网络在平方损失下对某类函数的表示结果，证实了深度网络可以更好地表示不太光滑的函数，其主要技术创新是充分利用深层网络可以用少量激活函数产生高振荡函数的事实。