We propose a new approach to learned optimization where we represent the
computation of an optimizer's update step using a neural network. The
parameters of the optimizer are then learned by training on a set of
optimization tasks with the objective to perform minimization efficiently. Our
innovation is a new neural network architecture, Optimus, for the learned
optimizer inspired by the classic BFGS algorithm. As in BFGS, we estimate a
preconditioning matrix as a sum of rank-one updates but use a Transformer-based
neural network to predict these updates jointly with the step length and
direction. In contrast to several recent learned optimization-based approaches,
our formulation allows for conditioning across the dimensions of the parameter
space of the target problem while remaining applicable to optimization tasks of
variable dimensionality without retraining. We demonstrate the advantages of
our approach on a benchmark composed of objective functions traditionally used
for the evaluation of optimization algorithms, as well as on the real
world-task of physics-based visual reconstruction of articulated 3d human
motion.

本文提出了一种新的学习优化方法，其中采用神经网络表示优化器更新步骤的计算，优化器的参数通过训练一组优化任务来学习，以实现高效的最小化。创新之处在于，它是一种新型的神经网络架构，“Optimus”，对学习到的优化器进行了改进，启发自经典的 BFGS 算法。

基于 Transformer 的学习优化

Transformer-Based Learned Optimization

Fitting parametric models of human bodies, hands or faces to sparse input
signals in an accurate, robust, and fast manner has the promise of
significantly improving immersion in AR and VR scenarios. A common first step
in systems that tackle these problems is to regress the parameters of the
parametric model directly from the input data. This approach is fast, robust,
and is a good starting point for an iterative minimization algorithm. The
latter searches for the minimum of an energy function, typically composed of a
data term and priors that encode our knowledge about the problem's structure.
While this is undoubtedly a very successful recipe, priors are often hand
defined heuristics and finding the right balance between the different terms to
achieve high quality results is a non-trivial task. Furthermore, converting and
optimizing these systems to run in a performant way requires custom
implementations that demand significant time investments from both engineers
and domain experts. In this work, we build upon recent advances in learned
optimization and propose an update rule inspired by the classic
Levenberg-Marquardt algorithm. We show the effectiveness of the proposed neural
optimizer on three problems, 3D body estimation from a head-mounted device, 3D
body estimation from sparse 2D keypoints and face surface estimation from dense
2D landmarks. Our method can easily be applied to new model fitting problems
and offers a competitive alternative to well-tuned 'traditional' model fitting
pipelines, both in terms of accuracy and speed.

通过学习优化方法，提出了一种类 Levenberg-Marquardt 算法的更新规则，以更高效地适应新的模型拟合问题，并且在准确性和速度上具有竞争力。

学习拟合形态模型

Learning to Fit Morphable Models

We analyze the type of learned optimization that occurs when a learned model
(such as a neural network) is itself an optimizer - a situation we refer to as
mesa-optimization, a neologism we introduce in this paper. We believe that the
possibility of mesa-optimization raises two important questions for the safety
and transparency of advanced machine learning systems. First, under what
circumstances will learned models be optimizers, including when they should not
be? Second, when a learned model is an optimizer, what will its objective be -
how will it differ from the loss function it was trained under - and how can it
be aligned? In this paper, we provide an in-depth analysis of these two primary
questions and provide an overview of topics for future research.

本文分析了当学习模型（例如神经网络）本身是优化器时所发生的学习优化类型，称之为 mesa-optimization。我们认为，mesa-optimization 的可能性对于先进机器学习系统的安全性和透明度提出了两个重要问题。第一，什么情况下学习模型会成为优化器，包括不应成为优化器的情况？第二，当学习模型是优化器时，它的目标将是什么，它将如何不同于它所训练的损失函数，如何进行对齐？本文对这两个主要问题进行了深入分析，并提供了未来研究的主题概述。