Residual networks (ResNets) have significantly better trainability and thus
performance than feed-forward networks at large depth. Introducing skip
connections facilitates signal propagation to deeper layers. In addition,
previous works found that adding a scaling parameter for the residual branch
further improves generalization performance. While they empirically identified
a particularly beneficial range of values for this scaling parameter, the
associated performance improvement and its universality across network
hyperparameters yet need to be understood. For feed-forward networks (FFNets),
finite-size theories have led to important insights with regard to signal
propagation and hyperparameter tuning. We here derive a systematic finite-size
theory for ResNets to study signal propagation and its dependence on the
scaling for the residual branch. We derive analytical expressions for the
response function, a measure for the network's sensitivity to inputs, and show
that for deep networks the empirically found values for the scaling parameter
lie within the range of maximal sensitivity. Furthermore, we obtain an
analytical expression for the optimal scaling parameter that depends only
weakly on other network hyperparameters, such as the weight variance, thereby
explaining its universality across hyperparameters. Overall, this work provides
a framework for theory-guided optimal scaling in ResNets and, more generally,
provides the theoretical framework to study ResNets at finite widths.

通过有限尺寸理论，研究残差网络的信号传播及其依赖残差分支的伸缩，发现最优伸缩参数范围在最大灵敏度范围内，并给出一个理论框架指导 ResNets 的最优伸缩。