Recent advancements in meta-learning have enabled the automatic discovery of
novel reinforcement learning algorithms parameterized by surrogate objective
functions. To improve upon manually designed algorithms, the parameterization
of this learned objective function must be expressive enough to represent novel
principles of learning (instead of merely recovering already established ones)
while still generalizing to a wide range of settings outside of its
meta-training distribution. However, existing methods focus on discovering
objective functions that, like many widely used objective functions in
reinforcement learning, do not take into account the total number of steps
allowed for training, or "training horizon". In contrast, humans use a plethora
of different learning objectives across the course of acquiring a new ability.
For instance, students may alter their studying techniques based on the
proximity to exam deadlines and their self-assessed capabilities. This paper
contends that ignoring the optimization time horizon significantly restricts
the expressive potential of discovered learning algorithms. We propose a simple
augmentation to two existing objective discovery approaches that allows the
discovered algorithm to dynamically update its objective function throughout
the agent's training procedure, resulting in expressive schedules and increased
generalization across different training horizons. In the process, we find that
commonly used meta-gradient approaches fail to discover such adaptive objective
functions while evolution strategies discover highly dynamic learning rules. We
demonstrate the effectiveness of our approach on a wide range of tasks and
analyze the resulting learned algorithms, which we find effectively balance
exploration and exploitation by modifying the structure of their learning rules
throughout the agent's lifetime.

最近的元学习进展使得可以自动发现由代理目标函数参数化的新型强化学习算法。本文提出一种对两种现有目标发现方法进行简单扩展的方案，允许在智能体的训练过程中动态更新其目标函数，从而获得具有表达能力的进度表，并增加在不同训练时间范围内的泛化能力。