The well-designed structures in neural networks reflect the prior knowledge
incorporated into the models. However, though different models have various
priors, we are used to training them with model-agnostic optimizers such as
SGD. In this paper, we propose to incorporate model-specific prior knowledge
into optimizers by modifying the gradients according to a set of model-specific
hyper-parameters. Such a methodology is referred to as Gradient
Re-parameterization, and the optimizers are named RepOptimizers. For the
extreme simplicity of model structure, we focus on a VGG-style plain model and
showcase that such a simple model trained with a RepOptimizer, which is
referred to as RepOpt-VGG, performs on par with or better than the recent
well-designed models. From a practical perspective, RepOpt-VGG is a favorable
base model because of its simple structure, high inference speed and training
efficiency. Compared to Structural Re-parameterization, which adds priors into
models via constructing extra training-time structures, RepOptimizers require
no extra forward/backward computations and solve the problem of quantization.
We hope to spark further research beyond the realms of model structure design.
Code and models https://github.com/DingXiaoH/RepOptimizers.

本研究通过梯度重新参数化的方法设计了 RepOptimizers 优化器，将模型特定的先验知识融入最优化器中。在 VGG 风格的平面模型上的实验表明，使用 RepOptimizers 的简单模型性能不逊于复杂的设计模型，并且具有更快的推理速度和训练效率。