This article derives and validates three principles for initialization and
architecture selection in finite width graph neural networks (GNNs) with ReLU
activations. First, we theoretically derive what is essentially the unique
generalization to ReLU GNNs of the well-known He-initialization. Our
initialization scheme guarantees that the average scale of network outputs and
gradients remains order one at initialization. Second, we prove in finite width
vanilla ReLU GNNs that oversmoothing is unavoidable at large depth when using
fixed aggregation operator, regardless of initialization. We then prove that
using residual aggregation operators, obtained by interpolating a fixed
aggregation operator with the identity, provably alleviates oversmoothing at
initialization. Finally, we show that the common practice of using residual
connections with a fixup-type initialization provably avoids correlation
collapse in final layer features at initialization. Through ablation studies we
find that using the correct initialization, residual aggregation operators, and
residual connections in the forward pass significantly and reliably speeds up
early training dynamics in deep ReLU GNNs on a variety of tasks.

本文提出了三个初始化和架构选择原则的验证，在有限宽度图神经网络（GNN）和 ReLU 激活下，通过证明得出使用 He-initialization，并在初始时使用残差聚合运算符和残差连接的实践可以显著提高深度 ReLU GNN 在各种任务上的早期训练动力学。