A key challenge in lifelong reinforcement learning (RL) is the loss of
plasticity, where previous learning progress hinders an agent's adaptation to
new tasks. While regularization and resetting can help, they require precise
hyperparameter selection at the outset and environment-dependent adjustments.
Building on the principled theory of online convex optimization, we present a
parameter-free optimizer for lifelong RL, called PACE, which requires no tuning
or prior knowledge about the distribution shifts. Extensive experiments on
Procgen, Atari, and Gym Control environments show that PACE works surprisingly
well$\unicode{x2013}$mitigating loss of plasticity and rapidly adapting to
challenging distribution shifts$\unicode{x2013}$despite the underlying
optimization problem being nonconvex and nonstationary.

PACE 是一种无需超参数调整和先验知识的参数自由优化器，基于在线凸优化理论解决了终身强化学习中可塑性损失、适应新任务以及分布变化等挑战。实验证明，尽管底层优化问题是非凸和非平稳的，PACE 在 Procgen，Atari 和 Gym Control 环境中表现出色，有效缓解了可塑性损失并快速适应挑战性的分布变化。

捷足先登：一种无参数的终身强化学习优化器

Pick up the PACE: A Parameter-Free Optimizer for Lifelong Reinforcement  Learning

Hyperparameter tuning, particularly the selection of an appropriate learning
rate in adaptive gradient training methods, remains a challenge. To tackle this
challenge, in this paper, we propose a novel parameter-free optimizer, AdamG
(Adam with the golden step size), designed to automatically adapt to diverse
optimization problems without manual tuning. The core technique underlying
AdamG is our golden step size derived for the AdaGrad-Norm algorithm, which is
expected to help AdaGrad-Norm preserve the tuning-free convergence and
approximate the optimal step size in expectation w.r.t. various optimization
scenarios. To better evaluate tuning-free performance, we propose a novel
evaluation criterion, stability, to comprehensively assess the efficacy of
parameter-free optimizers in addition to classical performance criteria.
Empirical results demonstrate that compared with other parameter-free
baselines, AdamG achieves superior performance, which is consistently on par
with Adam using a manually tuned learning rate across various optimization
tasks.

提出了一种无需手动调节参数的优化器 AdamG，通过使用 AdaGrad-Norm 算法中的黄金步长派生技术，自动适应不同的优化问题并实现了与手动调节学习率的 Adam 相当的优化性能。