Stacking many layers to create truly deep neural networks is arguably what
has led to the recent explosion of these methods. However, many properties of
deep neural networks are not yet understood. One such mystery is the depth
degeneracy phenomenon: the deeper you make your network, the closer your
network is to a constant function on initialization. In this paper, we examine
the evolution of the angle between two inputs to a ReLU neural network as a
function of the number of layers. By using combinatorial expansions, we find
precise formulas for how fast this angle goes to zero as depth increases. Our
formulas capture microscopic fluctuations that are not visible in the popular
framework of infinite width limits, and yet have a significant effect on
predicted behaviour. The formulas are given in terms of the mixed moments of
correlated Gaussians passed through the ReLU function. We also find a
surprising combinatorial connection between these mixed moments and the Bessel
numbers.

论文研究神经网络深度退化现象和该现象与两输入的夹角随层数变化的关系，使用组合拓展方法得出随着层数增加该夹角趋近于零的精确公式，并发现这些公式与相关高斯函数的混合矩和贝塞尔数之间存在意外的组合连接。