Standard neural networks struggle to generalize under distribution shifts. For out-of-distribution generalization in computer vision, the best current approach averages the weights along a training run. In this paper, we propose Diverse Weight Averaging (DiWA) that makes a simple change to this strategy: DiWA averages the weights obtained from several independent training runs rather than from a single run. Perhaps surprisingly, averaging these weights performs well under soft constraints despite the network's nonlinearities. The main motivation behind DiWA is to increase the functional diversity across averaged models. Indeed, models obtained from different runs are more diverse than those collected along a single run thanks to differences in hyperparameters and training procedures. We motivate the need for diversity by a new bias-variance-covariance-locality decomposition of the expected error, exploiting similarities between DiWA and standard functional ensembling. Moreover, this decomposition highlights that DiWA succeeds when the variance term dominates, which we show happens when the marginal distribution changes at test time. Experimentally, DiWA consistently improves the state of the art on the competitive DomainBed benchmark without inference overhead.

本文介绍了一种新的权重平均策略——Diverse Weight Averaging (DiWA)，通过多次独立训练获得的模型权重进行平均，以增加模型的功能多样性，并提出了一种新的误差分解方法来说明多模型平均在测试期间建模分布发生变化时成功的原因，并通过实验证明，DiWA在不增加推理开销的情况下始终提高了 DomainBed 基准的表现。

多元加权平均用于区分分布外泛化