学习优化器的扩展是否值得？评估 VeLO 的价值 4000 TPU 月

Oct, 2023

学习优化器的扩展是否值得？评估 VeLO 的价值 4000 TPU 月

Is Scaling Learned Optimizers Worth It? Evaluating The Value of VeLO's 4000 TPU Months

Fady Rezk, Antreas Antoniou, Henry Gouk, Timothy Hospedales

TL;DR通过分析 VeLO（通用学习优化器）的训练效果，我们发现其具有关键的超参数调整问题，不一定能比其他竞争对手更好地找到解决方案，并且在降低训练损失方面也不一定比其他优化器更快。这些观察结果对 VeLO 的广泛应用和培训投资的价值提出了质疑。

Abstract

We analyze velo (versatile learned optimizer), the largest scale attempt to train a general purpose "foundational" optimizer to date. velo was trained on thousands of machine learning tasks using over 4000 TPU mo