This study unveils a cutting-edge technique for neural network pruning that
judiciously addresses noisy gradients during the computation of the empirical
Fisher Information Matrix (FIM). We introduce an entropic Wasserstein
regression (EWR) formulation, capitalizing on the geometric attributes of the
optimal transport (OT) problem. This is analytically showcased to excel in
noise mitigation by adopting neighborhood interpolation across data points. The
unique strength of the Wasserstein distance is its intrinsic ability to strike
a balance between noise reduction and covariance information preservation.
Extensive experiments performed on various networks show comparable performance
of the proposed method with state-of-the-art (SoTA) network pruning algorithms.
Our proposed method outperforms the SoTA when the network size or the target
sparsity is large, the gain is even larger with the existence of noisy
gradients, possibly from noisy data, analog memory, or adversarial attacks.
Notably, our proposed method achieves a gain of 6% improvement in accuracy and
8% improvement in testing loss for MobileNetV1 with less than one-fourth of the
network parameters remaining.

本研究提出了一种先进的神经网络剪枝技术，通过在计算经验 Fisher 信息矩阵（FIM）过程中巧妙处理嘈杂的梯度。我们介绍了一种基于几何属性的熵 Wasserstein 回归（EWR）模型，通过采用数据点之间的邻域插值实现噪声缓解。Wasserstein 距离的独特优势在于平衡噪声降低和协方差信息保留之间的关系。通过在各种网络上进行广泛实验证明，所提出的方法在网络剪枝算法中具有与先进方法相当的性能。当网络规模或目标稀疏度较大时，我们提出的方法在存在嘈杂梯度的情况下效果更为显著，可能来自嘈杂数据、模拟存储器或对抗攻击。值得注意的是，我们的方法在仅剩不到四分之一的网络参数的情况下，使 MobileNetV1 的准确度提高了 6%，测试损失提高了 8%。