Deep neural networks often suffer from poor generalization due to complex and
non-convex loss landscapes. Sharpness-Aware Minimization (SAM) is a popular
solution that smooths the loss landscape by minimizing the maximized change of
training loss when adding a perturbation to the weight. However, indiscriminate
perturbation of SAM on all parameters is suboptimal and results in excessive
computation, double the overhead of common optimizers like Stochastic Gradient
Descent (SGD). In this paper, we propose Sparse SAM (SSAM), an efficient and
effective training scheme that achieves sparse perturbation by a binary mask.
To obtain the sparse mask, we provide two solutions based on Fisher information
and dynamic sparse training, respectively. We investigate the impact of
different masks, including unstructured, structured, and $N$:$M$ structured
patterns, as well as explicit and implicit forms of implementing sparse
perturbation. We theoretically prove that SSAM can converge at the same rate as
SAM, i.e., $O(\log T/\sqrt{T})$. Sparse SAM has the potential to accelerate
training and smooth the loss landscape effectively. Extensive experimental
results on CIFAR and ImageNet-1K confirm that our method is superior to SAM in
terms of efficiency, and the performance is preserved or even improved with a
perturbation of merely 50\% sparsity. Code is available at
this https URL

本文提出一种高效的 Sparse SAM 训练 scheme 来有效减少训练中的计算量，以达到平滑损失函数的目标，并提供了基于 Fisher information 和 dynamic sparse training 的两种不同的稀疏 mask 解决方案来实现稀疏干扰，理论证明了与 SAM 相同的收敛速度和有效性，并在 CIFAR 和 ImageNet-1K 上的实验证明了与 SAM 相比具有更好的效率和表现。

稀疏扰动锐度感知极小化优化器的系统研究

Systematic Investigation of Sparse Perturbed Sharpness-Aware  Minimization Optimizer

Deep neural networks often suffer from poor generalization caused by complex
and non-convex loss landscapes. One of the popular solutions is Sharpness-Aware
Minimization (SAM), which smooths the loss landscape via minimizing the
maximized change of training loss when adding a perturbation to the weight.
However, we find the indiscriminate perturbation of SAM on all parameters is
suboptimal, which also results in excessive computation, i.e., double the
overhead of common optimizers like Stochastic Gradient Descent (SGD). In this
paper, we propose an efficient and effective training scheme coined as Sparse
SAM (SSAM), which achieves sparse perturbation by a binary mask. To obtain the
sparse mask, we provide two solutions which are based onFisher information and
dynamic sparse training, respectively. In addition, we theoretically prove that
SSAM can converge at the same rate as SAM, i.e., $O(\log T/\sqrt{T})$. Sparse
SAM not only has the potential for training acceleration but also smooths the
loss landscape effectively. Extensive experimental results on CIFAR10,
CIFAR100, and ImageNet-1K confirm the superior efficiency of our method to SAM,
and the performance is preserved or even better with a perturbation of merely
50% sparsity. Code is availiable at
this https URL.

本文介绍了 Sparse SAM 的有效训练方案，通过使用二进制掩码获取稀疏掩码，基于 Fisher 信息和动态稀疏训练提供了两种解决方案，理论上证明了 Sparse SAM 可以以相同的速度收敛，既有潜力加速训练，又可以有效平滑损失地形。