Despite the simplicity, stochastic gradient descent (SGD)-like algorithms are
successful in training deep neural networks (DNNs). Among various attempts to
improve SGD, weight averaging (WA), which averages the weights of multiple
models, has recently received much attention in the literature. Broadly, WA
falls into two categories: 1) online WA, which averages the weights of multiple
models trained in parallel, is designed for reducing the gradient communication
overhead of parallel mini-batch SGD, and 2) offline WA, which averages the
weights of one model at different checkpoints, is typically used to improve the
generalization ability of DNNs. Though online and offline WA are similar in
form, they are seldom associated with each other. Besides, these methods
typically perform either offline parameter averaging or online parameter
averaging, but not both. In this work, we firstly attempt to incorporate online
and offline WA into a general training framework termed Hierarchical Weight
Averaging (HWA). By leveraging both the online and offline averaging manners,
HWA is able to achieve both faster convergence speed and superior
generalization performance without any fancy learning rate adjustment. Besides,
we also analyze the issues faced by existing WA methods, and how our HWA
address them, empirically. Finally, extensive experiments verify that HWA
outperforms the state-of-the-art methods significantly.

本文介绍了一种名为 Hierarchical Weight Averaging (HWA) 的新的深度学习神经网络权重平均训练框架，通过整合在线和离线平均方法，它能够同时提高收敛速度和泛化性能，同时解决了现有 WA 方法面临的问题。实验证明，HWA 明显优于现有方法。