Ensemble models often improve generalization performances in challenging
tasks. Yet, traditional techniques based on prediction averaging incur three
well-known disadvantages: the computational overhead of training multiple
models, increased latency, and memory requirements at test time. To address
these issues, the Stochastic Weight Averaging (SWA) technique maintains a
running average of model parameters from a specific epoch onward. Despite its
potential benefits, maintaining a running average of parameters can hinder
generalization, as an underlying running model begins to overfit. Conversely,
an inadequately chosen starting point can render SWA more susceptible to
underfitting compared to an underlying running model. In this work, we propose
Adaptive Stochastic Weight Averaging (ASWA) technique that updates a running
average of model parameters, only when generalization performance is improved
on the validation dataset. Hence, ASWA can be seen as a combination of SWA with
the early stopping technique, where the former accepts all updates on a
parameter ensemble model and the latter rejects any update on an underlying
running model. We conducted extensive experiments ranging from image
classification to multi-hop reasoning over knowledge graphs. Our experiments
over 11 benchmark datasets with 7 baseline models suggest that ASWA leads to a
statistically better generalization across models and datasets

提出了自适应随机权重平均（ASWA）技术，该技术结合了随机权重平均（SWA）和提前停止技术，仅在验证数据集上提高泛化性能时更新模型参数的运行平均值。对于图像分类到知识图谱上的多跳推理等广泛实验中，结果表明 ASWA 在模型和数据集上都能实现统计上更好的泛化。