Undeniably, Large Language Models (LLMs) have stirred an extraordinary wave
of innovation in the machine learning research domain, resulting in substantial
impact across diverse fields such as reinforcement learning, robotics, and
computer vision. Their incorporation has been rapid and transformative, marking
a significant paradigm shift in the field of machine learning research.
However, the field of experimental design, grounded on black-box
optimization, has been much less affected by such a paradigm shift, even though
integrating LLMs with optimization presents a unique landscape ripe for
exploration. In this position paper, we frame the field of black-box
optimization around sequence-based foundation models and organize their
relationship with previous literature. We discuss the most promising ways
foundational language models can revolutionize optimization, which include
harnessing the vast wealth of information encapsulated in free-form text to
enrich task comprehension, utilizing highly flexible sequence models such as
Transformers to engineer superior optimization strategies, and enhancing
performance prediction over previously unseen search spaces.

庞大的语言模型 (Large Language Models) 已经在机器学习研究领域引发了一股创新浪潮，在强化学习、机器人学和计算机视觉等多个领域产生了实质性影响。然而，基于黑箱优化的实验设计领域受到的这种范式转变影响较小，尽管将 LLM 与优化相结合呈现出一个独特而广泛的探索空间。本文呈现了关于基于序列的基础模型构成黑箱优化领域的框架，并组织了这些模型与先前文献的关系。我们讨论了基础语言模型能够革新优化的最有前景的方式，包括利用自由文本中蕴含的大量信息来丰富任务理解、利用高度灵活的序列模型例如 Transformers 来设计出更优的优化策略，并提高对以前未见的搜索空间的性能预测。