In modern deep learning, the models are learned by applying gradient updates using an optimizer, which transforms the updates based on various statistics. Optimizers are often hand-designed and tuning their hyperparameters is a big part of the training process. Learned optimizers have shown some initial promise, but are generally unsuccessful as a general optimization mechanism applicable to every problem. In this work we explore a different direction: instead of learning general optimizers, we instead specialize them to a specific training environment. We propose a novel optimizer technique that learns a layer-specific linear combination of update directions provided by a set of base optimizers, effectively adapting its strategy to the specific model and dataset. When evaluated on image classification tasks, this specialized optimizer significantly outperforms both traditional off-the-shelf methods such as Adam, as well as existing general learned optimizers. Moreover, it demonstrates robust generalization with respect to model initialization, evaluating on unseen datasets, and training durations beyond its meta-training horizon.

本研究解决了现有学习优化器在普适性上的不足，通过针对特定训练环境专门设计优化器，提出了一种新的优化技术，该技术能够根据基础优化器提供的更新方向，学习层特定的线性组合。在图像分类任务中，这种专门化优化器显著优于传统的优化方法，如Adam，并且在模型初始化、未见数据集和超出元训练范围的训练时长方面表现出强大的泛化能力。

聚焦于特定领域：针对预训练模型的学习优化器