L$_2$ regularization and weight decay regularization are equivalent for
standard stochastic gradient descent (when rescaled by the learning rate), but
as we demonstrate this is \emph{not} the case for adaptive gradient algorithms,
such as Adam. While common implementations of these algorithms employ L$_2$
regularization (often calling it "weight decay" in what may be misleading due
to the inequivalence we expose), we propose a simple modification to recover
the original formulation of weight decay regularization by \emph{decoupling}
the weight decay from the optimization steps taken w.r.t. the loss function. We
provide empirical evidence that our proposed modification (i) decouples the
optimal choice of weight decay factor from the setting of the learning rate for
both standard SGD and Adam and (ii) substantially improves Adam's
generalization performance, allowing it to compete with SGD with momentum on
image classification datasets (on which it was previously typically
outperformed by the latter). Our proposed decoupled weight decay has already
been adopted by many researchers, and the community has implemented it in
TensorFlow and PyTorch; the complete source code for our experiments is
available at this https URL

L$_2$ 正则化与权重衰减正则化在标准随机梯度下降中是等价的，但是在自适应梯度算法，比如 Adam 中并不相同。本文通过 “解耦” 权重衰减与代价函数的优化步骤，提出了一个简单的修改，从而恢复了原始的权重衰减规则。实验证据表明我们提出的修改不仅能够使得标准 SGD 和 Adam 中的权重衰减因素的最优选择与学习率的设置相分离，还能够显著提高 Adam 的泛化性能，从而使得它在图像分类数据集中可以与 SGD with momentum 竞争。