This paper aims to theoretically analyze the complexity of feature transformations encoded in DNNs with ReLU layers. We propose metrics to measure three types of complexities of transformations based on the information theory. We further discover and prove the strong correlation between the complexity and the disentanglement of transformations. Based on the proposed metrics, we analyze two typical phenomena of the change of the transformation complexity during the training process, and explore the ceiling of a DNN's complexity. The proposed metrics can also be used as a loss to learn a DNN with the minimum complexity, which also controls the over-fitting level of the DNN and influences adversarial robustness, adversarial transferability, and knowledge consistency. Comprehensive comparative studies have provided new perspectives to understand the DNN.

本文提出了一种使用信息论度量转换复杂度的指标，并探究了转换复杂度与 disentanglement 之间的强相关性，进一步分析了 DNN 训练过程中的两个典型现象，还探讨了 DNN 复杂度的上限，并使用提出的指标作为损失，学习具有最小复杂度的 DNN，并控制过拟合水平等，从而影响对抗鲁棒性、对抗传递性和知识一致性，同时提供了新的理解 DNN 的视角。

针对ReLU DNN的变换复杂度的理论分析