Learning to learn has emerged as an important direction for achieving artificial intelligence. Two of the primary barriers to its adoption are an inability to scale to larger problems and a limited ability to generalize to new tasks. We introduce a learned gradient descent optimizer that generalizes well to new tasks, and which has significantly reduced memory and computation overhead. We achieve this by introducing a novel hierarchical RNN architecture, with minimal per-parameter overhead, augmented with additional architectural features that mirror the known structure of optimization tasks. We also develop a meta-training ensemble of small, diverse, optimization tasks capturing common properties of loss landscapes. The optimizer learns to out-perform RMSProp/ADAM on problems in this corpus. More importantly, it performs comparably or better when applied to small convolutional neural networks, despite seeing no neural networks in its meta-training set. Finally, it generalizes to train Inception V3 and ResNet V2 architectures on the ImageNet dataset, optimization problems that are of a vastly different scale than those it was trained on.

通过引入层次循环神经网络优化算法和基于元学习的小任务集，实现了一个新的学习梯度下降优化器，解决了在更大的问题上扩展能力不足和泛化能力受限问题，并在ImageNet数据集上通过数千步为Inception V3和ResNet V2架构进行了优化。

可扩展且泛化的学习优化器