We propose a new perspective on representation learning in reinforcement
learning based on geometric properties of the space of value functions. We
leverage this perspective to provide formal evidence regarding the usefulness
of value functions as auxiliary tasks. Our formulation considers adapting the
representation to minimize the (linear) approximation of the value function of
all stationary policies for a given environment. We show that this optimization
reduces to making accurate predictions regarding a special class of value
functions which we call adversarial value functions (AVFs). We demonstrate that
using value functions as auxiliary tasks corresponds to an expected-error
relaxation of our formulation, with AVFs a natural candidate, and identify a
close relationship with proto-value functions (Mahadevan, 2005). We highlight
characteristics of AVFs and their usefulness as auxiliary tasks in a series of
experiments on the four-room domain.

通过基于值函数空间的几何特性，提出了一种新的表征学习的视角，证明了值函数作为辅助任务的实用性并将敌对价值函数作为其自然候选。在四间房间领域的实验中，展示了敌对价值函数作为辅助任务的有用性和特点。