Evolutionary optimization algorithms are often derived from loose biological
analogies and struggle to leverage information obtained during the sequential
course of optimization. An alternative promising approach is to leverage data
and directly discover powerful optimization principles via meta-optimization.
In this work, we follow such a paradigm and introduce Evolution Transformer, a
causal Transformer architecture, which can flexibly characterize a family of
Evolution Strategies. Given a trajectory of evaluations and search distribution
statistics, Evolution Transformer outputs a performance-improving update to the
search distribution. The architecture imposes a set of suitable inductive
biases, i.e. the invariance of the distribution update to the order of
population members within a generation and equivariance to the order of the
search dimensions. We train the model weights using Evolutionary Algorithm
Distillation, a technique for supervised optimization of sequence models using
teacher algorithm trajectories. The resulting model exhibits strong in-context
optimization performance and shows strong generalization capabilities to
otherwise challenging neuroevolution tasks. We analyze the resulting properties
of the Evolution Transformer and propose a technique to fully
self-referentially train the Evolution Transformer, starting from a random
initialization and bootstrapping its own learning progress. We provide an open
source implementation under this https URL

通过元优化的方式，引入一种能够灵活表征一系列进化策略的因果 Transformer 架构 ——Evolution Transformer，用于从数据中直接发现强大的优化原理。通过 Evolution Transformer，结合进化算法蒸馏技术进行模型权重训练，我们得到一个在特定上下文中表现良好且能够很好泛化到其他具有挑战性的神经进化任务的模型。此外，我们还分析了 Evolution Transformer 的属性，并提出了一种全自引用训练的技术，从随机初始化开始并引导其自身的学习进程。我们提供了一个开源实现，网址为 https://<URL>