Structural re-parameterization (Rep) methods achieve noticeable improvements
on simple VGG-style networks. Despite the prevalence, current Rep methods
simply re-parameterize all operations into an augmented network, including
those that rarely contribute to the model's performance. As such, the price to
pay is an expensive computational overhead to manipulate these unnecessary
behaviors. To eliminate the above caveats, we aim to bootstrap the training
with minimal cost by devising a dynamic re-parameterization (DyRep) method,
which encodes Rep technique into the training process that dynamically evolves
the network structures. Concretely, our proposal adaptively finds the
operations which contribute most to the loss in the network, and applies Rep to
enhance their representational capacity. Besides, to suppress the noisy and
redundant operations introduced by Rep, we devise a de-parameterization
technique for a more compact re-parameterization. With this regard, DyRep is
more efficient than Rep since it smoothly evolves the given network instead of
constructing an over-parameterized network. Experimental results demonstrate
our effectiveness, e.g., DyRep improves the accuracy of ResNet-18 by $2.04\%$
on ImageNet and reduces $22\%$ runtime over the baseline. Code is available at:
this https URL

本文提出了动态重参数化方法，通过优化计算效率、提升表征能力，采用 mini-batch 实现动态演化神经网络，并将其应用于 ResNet-18 模型上，结果表明：在 ImageNet 数据集上，相比基线模型，精度提升了 2.04％，运行时长降低了 22％。