The ability of a reinforcement learning (RL) agent to learn about many reward
functions at the same time has many potential benefits, such as the
decomposition of complex tasks into simpler ones, the exchange of information
between tasks, and the reuse of skills. We focus on one aspect in particular,
namely the ability to generalise to unseen tasks. Parametric generalisation
relies on the interpolation power of a function approximator that is given the
task description as input; one of its most common form are universal value
function approximators (UVFAs). Another way to generalise to new tasks is to
exploit structure in the RL problem itself. Generalised policy improvement
(GPI) combines solutions of previous tasks into a policy for the unseen task;
this relies on instantaneous policy evaluation of old policies under the new
reward function, which is made possible through successor features (SFs). Our
proposed universal successor features approximators (USFAs) combine the
advantages of all of these, namely the scalability of UVFAs, the instant
inference of SFs, and the strong generalisation of GPI. We discuss the
challenges involved in training a USFA, its generalisation properties and
demonstrate its practical benefits and transfer abilities on a large-scale
domain in which the agent has to navigate in a first-person perspective
three-dimensional environment.

我们提出了一种新型的普适继承特征逼近器，利用其可进行上下文下多种策略迁移与推论，提高了强化学习算法的鲁棒性和实用性，并在一款首人称三维环境导航应用上展示了其广泛适用性。