This study unveils a cutting-edge technique for neural network pruning that judiciously addresses noisy gradients during the computation of the empirical Fisher Information Matrix (FIM). We introduce an entropic Wasserstein regression (EWR) formulation, capitalizing on the geometric attributes of the optimal transport (OT) problem. This is analytically showcased to excel in noise mitigation by adopting neighborhood interpolation across data points. The unique strength of the Wasserstein distance is its intrinsic ability to strike a balance between noise reduction and covariance information preservation. Extensive experiments performed on various networks show comparable performance of the proposed method with state-of-the-art (SoTA) network pruning algorithms. Our proposed method outperforms the SoTA when the network size or the target sparsity is large, the gain is even larger with the existence of noisy gradients, possibly from noisy data, analog memory, or adversarial attacks. Notably, our proposed method achieves a gain of 6% improvement in accuracy and 8% improvement in testing loss for MobileNetV1 with less than one-fourth of the network parameters remaining.

本研究提出了一种先进的神经网络剪枝技术，通过在计算经验Fisher信息矩阵（FIM）过程中巧妙处理嘈杂的梯度。我们介绍了一种基于几何属性的熵Wasserstein回归（EWR）模型，通过采用数据点之间的邻域插值实现噪声缓解。Wasserstein距离的独特优势在于平衡噪声降低和协方差信息保留之间的关系。通过在各种网络上进行广泛实验证明，所提出的方法在网络剪枝算法中具有与先进方法相当的性能。当网络规模或目标稀疏度较大时，我们提出的方法在存在嘈杂梯度的情况下效果更为显著，可能来自嘈杂数据、模拟存储器或对抗攻击。值得注意的是，我们的方法在仅剩不到四分之一的网络参数的情况下，使MobileNetV1的准确度提高了6%，测试损失提高了8%。

稀疏熵-瓦瑟斯坦回归强大的网络剪枝