Weight Average (WA) is an active research topic due to its simplicity in
ensembling deep networks and the effectiveness in promoting generalization.
Existing weight average approaches, however, are often carried out along only
one training trajectory in a post-hoc manner (i.e., the weights are averaged
after the entire training process is finished), which significantly degrades
the diversity between networks and thus impairs the effectiveness in
ensembling. In this paper, inspired by weight average, we propose Lookaround, a
straightforward yet effective SGD-based optimizer leading to flatter minima
with better generalization. Specifically, Lookaround iterates two steps during
the whole training period: the around step and the average step. In each
iteration, 1) the around step starts from a common point and trains multiple
networks simultaneously, each on transformed data by a different data
augmentation, and 2) the average step averages these trained networks to get
the averaged network, which serves as the starting point for the next
iteration. The around step improves the functionality diversity while the
average step guarantees the weight locality of these networks during the whole
training, which is essential for WA to work. We theoretically explain the
superiority of Lookaround by convergence analysis, and make extensive
experiments to evaluate Lookaround on popular benchmarks including CIFAR and
ImageNet with both CNNs and ViTs, demonstrating clear superiority over
state-of-the-arts. Our code is available at
this https URL

本研究提出了 Lookaround 优化器，其通过迭代使用环绕步骤和平均步骤的方式训练多个网络，实现了促进网络多样性与提升泛化能力的效果，理论分析和实验表明该优化器优于现有方法。