In this study, we explore an innovative approach for neural network
optimization, focusing on the application of gradient sampling techniques,
similar to those in StochGradAdam, during the pruning process. Our primary
objective is to maintain high accuracy levels in pruned models, a critical
challenge in resource-limited scenarios. Our extensive experiments reveal that
models optimized with gradient sampling techniques are more effective at
preserving accuracy during pruning compared to those using traditional
optimization methods. This finding underscores the significance of gradient
sampling in facilitating robust learning and enabling networks to retain
crucial information even after substantial reduction in their complexity. We
validate our approach across various datasets and neural architectures,
demonstrating its broad applicability and effectiveness. The paper also delves
into the theoretical aspects, explaining how gradient sampling techniques
contribute to the robustness of models during pruning. Our results suggest a
promising direction for creating efficient neural networks that do not
compromise on accuracy, even in environments with constrained computational
resources.

本研究探讨了一种创新的神经网络优化方法，专注于在修剪过程中应用与 StochGradAdam 类似的梯度采样技术。我们的主要目标是在修剪模型时保持高精度水平，这是资源受限场景中的一个重要挑战。我们广泛的实验证明，使用梯度采样技术优化的模型在修剪过程中相对于使用传统优化方法的模型更有效地保持准确性。这一发现强调了梯度采样在促进稳健学习方面的重要性，使得网络即使在复杂度大幅减少后仍能保留关键信息。我们在各种数据集和神经网络结构上验证了我们的方法，展示了其广泛的适用性和有效性。该论文还深入研究了理论方面，解释了梯度采样技术在修剪过程中对模型稳健性的贡献。我们的结果为在计算资源受限的环境中创建不会牺牲准确性的高效神经网络指明了一个有希望的方向。