Bayesian optimization (BO) is a powerful approach for optimizing complex and
expensive-to-evaluate black-box functions. Its importance is underscored in
many applications, notably including hyperparameter tuning, but its efficacy
depends on efficiently balancing exploration and exploitation. While there has
been substantial progress in BO methods, striking this balance still remains a
delicate process. In this light, we present \texttt{LLAMBO}, a novel approach
that integrates the capabilities of large language models (LLM) within BO. At a
high level, we frame the BO problem in natural language terms, enabling LLMs to
iteratively propose promising solutions conditioned on historical evaluations.
More specifically, we explore how combining contextual understanding, few-shot
learning proficiency, and domain knowledge of LLMs can enhance various
components of model-based BO. Our findings illustrate that \texttt{LLAMBO} is
effective at zero-shot warmstarting, and improves surrogate modeling and
candidate sampling, especially in the early stages of search when observations
are sparse. Our approach is performed in context and does not require LLM
finetuning. Additionally, it is modular by design, allowing individual
components to be integrated into existing BO frameworks, or function cohesively
as an end-to-end method. We empirically validate \texttt{LLAMBO}'s efficacy on
the problem of hyperparameter tuning, highlighting strong empirical performance
across a range of diverse benchmarks, proprietary, and synthetic tasks.

LLAMBO 是一种将大型语言模型（LLM）与贝叶斯优化（BO）结合的方法，通过在自然语言中提出有前景的解决方案，利用上下文理解、少样本学习能力和 LLM 的领域知识来增强基于模型的 BO 的各个组件，特别是在稀疏的观测阶段，LLAMBO 在零样本热启动、代理模型和候选样本的改进方面表现出有效性。