BriefGPT.xyz
Dec, 2023
学习优化器的训练动态调查
Investigation into the Training Dynamics of Learned Optimizers
HTML
PDF
Jan Sobotka, Petr Šimánek, Daniel Vašata
TL;DR
深度学习中优化的关键问题是通过学习优化器来加速优化过程，但其稳定性、泛化性仍存在问题。本研究通过分析网络架构对优化轨迹和参数更新分布的影响，研究并对比手动设计和学习优化器的优缺点，提出了关键见解。
Abstract
Optimization is an integral part of modern
deep learning
. Recently, the concept of
learned optimizers
has emerged as a way to accelerate this optimization process by replacing traditional, hand-crafted algorithms
→