Reparameterization aims to improve the generalization of deep neural networks
by transforming convolutional layers into equivalent multi-branched structures
during training. However, there exists a gap in understanding how
reparameterization may change and benefit the learning process of neural
networks. In this paper, we present a novel spatial gradient scaling method to
redistribute learning focus among weights in convolutional networks. We prove
that spatial gradient scaling achieves the same learning dynamics as a branched
reparameterization yet without introducing structural changes into the network.
We further propose an analytical approach that dynamically learns scalings for
each convolutional layer based on the spatial characteristics of its input
feature map gauged by mutual information. Experiments on CIFAR-10, CIFAR-100,
and ImageNet show that without searching for reparameterized structures, our
proposed scaling method outperforms the state-of-the-art reparameterization
strategies at a lower computational cost.

本论文提出了一种新颖的空间梯度缩放方法，该方法根据特征映射的空间特征测量互信息，动态学习每个卷积层的缩放，并证明了空间梯度缩放实现了与分支再参数化相同的学习动态，同时不引入网络结构变化。实验结果表明，我们提出的缩放方法在不搜索再参数化结构的情况下，以较低的计算成本优于最先进的再参数化策略。