The sample efficiency of Bayesian optimization algorithms depends on
carefully crafted acquisition functions (AFs) guiding the sequential collection
of function evaluations. The best-performing AF can vary significantly across
optimization problems, often requiring ad-hoc and problem-specific choices.
This work tackles the challenge of designing novel AFs that perform well across
a variety of experimental settings. Based on FunSearch, a recent work using
Large Language Models (LLMs) for discovery in mathematical sciences, we propose
FunBO, an LLM-based method that can be used to learn new AFs written in
computer code by leveraging access to a limited number of evaluations for a set
of objective functions. We provide the analytic expression of all discovered
AFs and evaluate them on various global optimization benchmarks and
hyperparameter optimization tasks. We show how FunBO identifies AFs that
generalize well in and out of the training distribution of functions, thus
outperforming established general-purpose AFs and achieving competitive
performance against AFs that are customized to specific function types and are
learned via transfer-learning algorithms.

该论文提出了一种基于大型语言模型的方法 FunBO，可以学习用于函数优化的新的采集函数，并在各种全局优化基准和超参数优化任务中取得竞争性性能。

FunBO：基于 FunSearch 的贝叶斯优化的发现式获取函数

FunBO: Discovering Acquisition Functions for Bayesian Optimization with  FunSearch

High-dimensional problems have long been considered the Achilles' heel of
Bayesian optimization algorithms. Spurred by the curse of dimensionality, a
large collection of algorithms aim to make it more performant in this setting,
commonly by imposing various simplifying assumptions on the objective. In this
paper, we identify the degeneracies that make vanilla Bayesian optimization
poorly suited to high-dimensional tasks, and further show how existing
algorithms address these degeneracies through the lens of lowering the model
complexity. Moreover, we propose an enhancement to the prior assumptions that
are typical to vanilla Bayesian optimization algorithms, which reduces the
complexity to manageable levels without imposing structural restrictions on the
objective. Our modification - a simple scaling of the Gaussian process
lengthscale prior with the dimensionality - reveals that standard Bayesian
optimization works drastically better than previously thought in high
dimensions, clearly outperforming existing state-of-the-art algorithms on
multiple commonly considered real-world high-dimensional tasks.

高维问题一直被认为是贝叶斯优化算法的致命软肋。本文通过识别导致标准贝叶斯优化在高维任务中性能不佳的退化情况，并通过减少模型复杂性的角度探究现有算法如何应对这些退化情况。此外，我们提出了一种增强对标准贝叶斯优化算法中先验假设的修改方法，通过与维度进行简单缩放的高斯过程长度尺度先验，揭示了在高维情况下标准贝叶斯优化比以往认为的表现要好得多，明显优于多个常见的真实高维任务的现有最先进算法。