We propose a new approach to learned optimization where we represent the
computation of an optimizer's update step using a neural network. The
parameters of the optimizer are then learned by training on a set of
optimization tasks with the objective to perform minimization efficiently. Our
innovation is a new neural network architecture, Optimus, for the learned
optimizer inspired by the classic BFGS algorithm. As in BFGS, we estimate a
preconditioning matrix as a sum of rank-one updates but use a Transformer-based
neural network to predict these updates jointly with the step length and
direction. In contrast to several recent learned optimization-based approaches,
our formulation allows for conditioning across the dimensions of the parameter
space of the target problem while remaining applicable to optimization tasks of
variable dimensionality without retraining. We demonstrate the advantages of
our approach on a benchmark composed of objective functions traditionally used
for the evaluation of optimization algorithms, as well as on the real
world-task of physics-based visual reconstruction of articulated 3d human
motion.

本文提出了一种新的学习优化方法，其中采用神经网络表示优化器更新步骤的计算，优化器的参数通过训练一组优化任务来学习，以实现高效的最小化。创新之处在于，它是一种新型的神经网络架构，“Optimus”，对学习到的优化器进行了改进，启发自经典的 BFGS 算法。

基于 Transformer 的学习优化

Transformer-Based Learned Optimization

We provide a variable metric stochastic approximation theory. In doing so, we
provide a convergence theory for a large class of online variable metric
methods including the recently introduced online versions of the BFGS algorithm
and its limited-memory LBFGS variant. We also discuss the implications of our
results for learning from expert advice.

本文提出了可变度量随机逼近理论，其中包括了在线 BFGS 算法及其有限内存 LBFGS 变体，并讨论了这一结果对学习专家建议的影响。

变量度量随机逼近理论

Variable Metric Stochastic Approximation Theory

We extend the well-known BFGS quasi-Newton method and its memory-limited
variant LBFGS to the optimization of nonsmooth convex objectives. This is done
in a rigorous fashion by generalizing three components of BFGS to
subdifferentials: the local quadratic model, the identification of a descent
direction, and the Wolfe line search conditions. We prove that under some
technical conditions, the resulting subBFGS algorithm is globally convergent in
objective function value. We apply its memory-limited variant (subLBFGS) to
L_2-regularized risk minimization with the binary hinge loss. To extend our
algorithm to the multiclass and multilabel settings, we develop a new,
efficient, exact line search algorithm. We prove its worst-case time complexity
bounds, and show that our line search can also be used to extend a recently
developed bundle method to the multiclass and multilabel settings. We also
apply the direction-finding component of our algorithm to L_1-regularized risk
minimization with logistic loss. In all these contexts our methods perform
comparable to or better than specialized state-of-the-art solvers on a number
of publicly available datasets. An open source implementation of our algorithms
is freely available.

本研究将众所周知的 BFGS 拟牛顿方法及其内存限制变种 LBFGS 扩展到非平滑凸目标的优化，提出了 subBFGS 算法，其全局收敛，并使用其记忆限制变体（subLBFGS）来最小化 L2 正则化风险并开发了新的多类和多标签设置下的准确搜索算法。