Much as replacing hand-designed features with learned functions has
revolutionized how we solve perceptual tasks, we believe learned algorithms
will transform how we train models. In this work we focus on general-purpose
learned optimizers capable of training a wide variety of problems with no
user-specified hyperparameters. We introduce a new, neural network
parameterized, hierarchical optimizer with access to additional features such
as validation loss to enable automatic regularization. Most learned optimizers
have been trained on only a single task, or a small number of tasks. We train
our optimizers on thousands of tasks, making use of orders of magnitude more
compute, resulting in optimizers that generalize better to unseen tasks. The
learned optimizers not only perform well, but learn behaviors that are distinct
from existing first order optimizers. For instance, they generate update steps
that have implicit regularization and adapt as the problem hyperparameters
(e.g. batch size) or architecture (e.g. neural network width) change. Finally,
these learned optimizers show evidence of being useful for out of distribution
tasks such as training themselves from scratch.

本文讲述了利用神经网络参数化的层次优化器提供自动正则化的能力，从而可以训练多种问题的通用学习优化器，可适应超参数和架构的变化，且适用于自我训练等需要解决分布问题的任务。