Selecting an appropriate optimizer for a given problem is of major interest
for researchers and practitioners. Many analytical optimizers have been
proposed using a variety of theoretical and empirical approaches; however, none
can offer a universal advantage over other competitive optimizers. We are thus
motivated to study a new problem named Optimizer Amalgamation: how can we best
combine a pool of "teacher" optimizers into a single "student" optimizer that
can have stronger problem-specific performance? In this paper, we draw
inspiration from the field of "learning to optimize" to use a learnable
amalgamation target. First, we define three differentiable amalgamation
mechanisms to amalgamate a pool of analytical optimizers by gradient descent.
Then, in order to reduce variance of the amalgamation process, we also explore
methods to stabilize the amalgamation process by perturbing the amalgamation
target. Finally, we present experiments showing the superiority of our
amalgamated optimizer compared to its amalgamated components and learning to
optimize baselines, and the efficacy of our variance reducing perturbations.
Our code and pre-trained models are publicly available at
this http URL

研究如何将一个优化器池中的优化器合并为单个性能更强的学生优化器。定义三种不同的可微合并机制，并探索稳定合并过程的方法，进而展示所合并的优化器相对于其组件和学习优化基线更为优越，以及利用扰动减少方差的有效性。