We have identified a potential method for unifying first-order optimizers
through the use of variable Second-Moment Exponential Scaling(SMES). We begin
with back propagation, addressing classic phenomena such as gradient vanishing
and explosion, as well as issues related to dataset sparsity, and introduce the
theory of balance in optimization. Through this theory, we suggest that SGD and
adaptive optimizers can be unified under a broader inference, employing
variable moving exponential scaling to achieve a balanced approach within a
generalized formula for first-order optimizers. We conducted tests on some
classic datasets and networks to confirm the impact of different balance
coefficients on the overall training process.

我们通过使用变量的二阶指数尺度（SMES）的潜在方法，提出了一种统一一阶优化器的可能方法。我们从反向传播开始，解决经典现象（如梯度消失和梯度爆炸），以及与数据集稀疏性相关的问题，引入了优化中的平衡理论。通过这个理论，我们认为可以在更广泛的推论下统一 SGD 和自适应优化器，在第一阶优化器的广义公式中采用变量移动指数尺度来实现平衡方法。我们对一些经典数据集和网络进行了测试，以确认不同平衡系数对整体训练过程的影响。