Second-order training methods have better convergence properties than
gradient descent but are rarely used in practice for large-scale training due
to their computational overhead. This can be viewed as a hardware limitation
(imposed by digital computers). Here we show that natural gradient descent
(NGD), a second-order method, can have a similar computational complexity per
iteration to a first-order method, when employing appropriate hardware. We
present a new hybrid digital-analog algorithm for training neural networks that
is equivalent to NGD in a certain parameter regime but avoids prohibitively
costly linear system solves. Our algorithm exploits the thermodynamic
properties of an analog system at equilibrium, and hence requires an analog
thermodynamic computer. The training occurs in a hybrid digital-analog loop,
where the gradient and Fisher information matrix (or any other positive
semi-definite curvature matrix) are calculated at given time intervals while
the analog dynamics take place. We numerically demonstrate the superiority of
this approach over state-of-the-art digital first- and second-order training
methods on classification tasks and language model fine-tuning tasks.

我们展示了在大规模训练中，与一阶方法相比，具有更好收敛性质的二阶训练方法很少被使用，可能是由于计算开销过大。然而，我们通过使用适当的硬件设备，证明了某些参数区间内，自然梯度下降（NGD）这种二阶方法在每次迭代中可以具备与一阶方法类似的计算复杂度。我们提出了一种新的混合数字 - 模拟算法用于神经网络训练，该算法在某些参数范围内等效于 NGD，但避免了解线性系统代价昂贵的求解。我们的算法利用了模拟系统在热力学平衡下的性质，因此需要模拟热力学计算机。在混合数字 - 模拟循环中进行训练，梯度和费舍尔信息矩阵（或任何正定曲率矩阵）在给定的时间间隔内计算，同时进行模拟动力学。在分类任务和语言模型微调任务上，我们通过数值实验证明了这种方法相对于最先进的数字一阶和二阶训练方法的优越性。