Computing the loss gradient via backpropagation consumes considerable energy
during deep learning (DL) model training. In this paper, we propose a novel
approach to efficiently compute DL models' gradients to mitigate the
substantial energy overhead associated with backpropagation. Exploiting the
over-parameterized nature of DL models and the smoothness of their loss
landscapes, we propose a method called {\em GradSamp} for sampling gradient
updates from a Gaussian distribution. Specifically, we update model parameters
at a given epoch (chosen periodically or randomly) by perturbing the parameters
(element-wise) from the previous epoch with Gaussian ``noise''. The parameters
of the Gaussian distribution are estimated using the error between the model
parameter values from the two previous epochs. {\em GradSamp} not only
streamlines gradient computation but also enables skipping entire epochs,
thereby enhancing overall efficiency. We rigorously validate our hypothesis
across a diverse set of standard and non-standard CNN and transformer-based
models, spanning various computer vision tasks such as image classification,
object detection, and image segmentation. Additionally, we explore its efficacy
in out-of-distribution scenarios such as Domain Adaptation (DA), Domain
Generalization (DG), and decentralized settings like Federated Learning (FL).
Our experimental results affirm the effectiveness of {\em GradSamp} in
achieving notable energy savings without compromising performance, underscoring
its versatility and potential impact in practical DL applications.

基于模型的过参数化性质和损失函数的平滑性，我们提出了一种名为 “GradSamp” 的方法，通过从高斯分布中采样梯度更新，以高效计算深度学习模型的梯度，从而减少反向传播过程中的能量开销。我们的实验结果验证了 “GradSamp” 在不损失性能的情况下，显著提高了能量效率，并表明其在实际深度学习应用中的多样性和潜在影响。