We introduce a novel optimization problem formulation that departs from the
conventional way of minimizing machine learning model loss as a black-box
function. Unlike traditional formulations, the proposed approach explicitly
incorporates an initially pre-trained model and random sketch operators,
allowing for sparsification of both the model and gradient during training. We
establish insightful properties of the proposed objective function and
highlight its connections to the standard formulation. Furthermore, we present
several variants of the Stochastic Gradient Descent (SGD) method adapted to the
new problem formulation, including SGD with general sampling, a distributed
version, and SGD with variance reduction techniques. We achieve tighter
convergence rates and relax assumptions, bridging the gap between theoretical
principles and practical applications, covering several important techniques
such as Dropout and Sparse training. This work presents promising opportunities
to enhance the theoretical understanding of model training through a
sparsification-aware optimization approach.

引入了一种新的优化问题形式，与传统的最小化机器学习模型损失的黑盒函数的方式不同。通过明确地纳入最初预训练模型和随机草图运算符，允许在训练过程中对模型和梯度进行稀疏化。本研究提出的目标函数具有深刻的性质，并强调其与标准算法的联系。同时，还介绍了几种适应新问题形式的随机梯度下降（SGD）方法的变种，包括具有一般抽样的 SGD、分布式版本和具有方差减小技术的 SGD。通过这种面向稀疏化的优化方法，实现了更紧凑的收敛速度和放松了假设，填补了理论原则与实际应用之间的差距，涵盖了 Dropout 和稀疏训练等几种重要技术。该研究为通过稀疏化感知的优化方法增强对模型训练的理论理解提供了有希望的机会。