We introduce a novel machine learning optimizer called LODO, which online meta-learns an implicit inverse Hessian of the loss as a subroutine of quasi-Newton optimization. Our optimizer merges Learning to Optimize (L2O) techniques with quasi-Newton methods to learn neural representations of symmetric matrix vector products, which are more flexible than those in other quasi-Newton methods. Unlike other L2O methods, ours does not require any meta-training on a training task distribution, and instead learns to optimize on the fly while optimizing on the test task, adapting to the local characteristics of the loss landscape while traversing it. Theoretically, we show that our optimizer approximates the inverse Hessian in noisy loss landscapes and is capable of representing a wide range of inverse Hessians. We experimentally verify our algorithm's performance in the presence of noise, and show that simpler alternatives for representing the inverse Hessians worsen performance. Lastly, we use our optimizer to train a semi-realistic deep neural network with 95k parameters, and obtain competitive results against standard neural network optimizers.

本文提出了一种新的机器学习优化器LODO，它将学习优化(L2O)技术与拟牛顿方法相结合，用于学习对称矩阵向量积的神经表示，从而适应于在测试任务中遍历的损失景观的局部特征。与其他L2O方法不同的是，我们的方法不需要在训练任务分布上进行任何元训练，并验证了其在噪声中的表现，并证明其能够表示一种广泛的逆Hessian。实验表明，简单的替代方法会导致性能变差。最后，我们使用我们的优化器训练一个拥有95k参数的半真实深度神经网络，并获得了与标准神经网络优化器竞争的结果。

学习优化拟牛顿方法