With the increasing computational costs associated with deep learning, automated hyperparameter optimization methods, strongly relying on black-box Bayesian optimization (BO), face limitations. Freeze-thaw BO offers a promising grey-box alternative, strategically allocating scarce resources incrementally to different configurations. However, the frequent surrogate model updates inherent to this approach pose challenges for existing methods, requiring retraining or fine-tuning their neural network surrogates online, introducing overhead, instability, and hyper-hyperparameters. In this work, we propose FT-PFN, a novel surrogate for Freeze-thaw style BO. FT-PFN is a prior-data fitted network (PFN) that leverages the transformers' in-context learning ability to efficiently and reliably do Bayesian learning curve extrapolation in a single forward pass. Our empirical analysis across three benchmark suites shows that the predictions made by FT-PFN are more accurate and 10-100 times faster than those of the deep Gaussian process and deep ensemble surrogates used in previous work. Furthermore, we show that, when combined with our novel acquisition mechanism (MFPI-random), the resulting in-context freeze-thaw BO method (ifBO), yields new state-of-the-art performance in the same three families of deep learning HPO benchmarks considered in prior work.

本文提出了一种名为FT-PFN的新型先验数据拟合网络，它利用了transformer在上下文学习方面的能力，以单次前向传播的方式高效可靠地进行贝叶斯学习曲线推断。经验证实，FT-PFN的预测比之前工作中使用的深高斯过程和深度集成模型更准确且速度快10-100倍，并结合新颖的采样机制（MFPI-random）应用于深度学习超参数优化任务中，取得了新的同类任务的最好性能。

上下文冷热贝叶斯优化：超参数优化