Large Transformer models are capable of implementing a plethora of so-called
in-context learning algorithms. These include gradient descent, classification,
sequence completion, transformation, and improvement. In this work, we
investigate whether large language models (LLMs), which never explicitly
encountered the task of black-box optimization, are in principle capable of
implementing evolutionary optimization algorithms. While previous works have
solely focused on language-based task specification, we move forward and focus
on the zero-shot application of LLMs to black-box optimization. We introduce a
novel prompting strategy, consisting of least-to-most sorting of discretized
population members and querying the LLM to propose an improvement to the mean
statistic, i.e. perform a type of black-box recombination operation.
Empirically, we find that our setup allows the user to obtain an LLM-based
evolution strategy, which we call `EvoLLM', that robustly outperforms baseline
algorithms such as random search and Gaussian Hill Climbing on synthetic BBOB
functions as well as small neuroevolution tasks. Hence, LLMs can act as
`plug-in' in-context recombination operators. We provide several comparative
studies of the LLM's model size, prompt strategy, and context construction.
Finally, we show that one can flexibly improve EvoLLM's performance by
providing teacher algorithm information via instruction fine-tuning on
previously collected teacher optimization trajectories.

大型语言模型（LLMs）具备黑箱优化任务的潜力，可以通过 least-to-most 排序的提示策略提供演化优化算法，称为 EvoLLM，从而在合成 BBOB 函数和小规模神经进化任务上稳定优于传统算法。并通过对 LLM 的模型规模、提示策略和上下文构建的比较研究表明，通过对先前收集的教师优化轨迹进行指令微调，能够灵活地提高 EvoLLM 的性能。