The rank of neural networks measures information flowing across layers. It is
an instance of a key structural condition that applies across broad domains of
machine learning. In particular, the assumption of low-rank feature
representations leads to algorithmic developments in many architectures. For
neural networks, however, the intrinsic mechanism that yields low-rank
structures remains vague and unclear. To fill this gap, we perform a rigorous
study on the behavior of network rank, focusing particularly on the notion of
rank deficiency. We theoretically establish a universal monotonic decreasing
property of network rank from the basic rules of differential and algebraic
composition, and uncover rank deficiency of network blocks and deep function
coupling. By virtue of our numerical tools, we provide the first empirical
analysis of the per-layer behavior of network rank in practical settings, i.e.,
ResNets, deep MLPs, and Transformers on ImageNet. These empirical results are
in direct accord with our theory. Furthermore, we reveal a novel phenomenon of
independence deficit caused by the rank deficiency of deep networks, where
classification confidence of a given category can be linearly decided by the
confidence of a handful of other categories. The theoretical results of this
work, together with the empirical findings, may advance understanding of the
inherent principles of deep neural networks.

本文研究神经网络的秩与结构缺陷，通过理论和实验发现了深度神经网络中秩缺陷和分类的独立性缺陷。