In reinforcement learning (RL), state representations are key to dealing with
large or continuous state spaces. While one of the promises of deep learning
algorithms is to automatically construct features well-tuned for the task they
try to solve, such a representation might not emerge from end-to-end training
of deep RL agents. To mitigate this issue, auxiliary objectives are often
incorporated into the learning process and help shape the learnt state
representation. Bootstrapping methods are today's method of choice to make
these additional predictions. Yet, it is unclear which features these
algorithms capture and how they relate to those from other auxiliary-task-based
approaches. In this paper, we address this gap and provide a theoretical
characterization of the state representation learnt by temporal difference
learning (Sutton, 1988). Surprisingly, we find that this representation differs
from the features learned by Monte Carlo and residual gradient algorithms for
most transition structures of the environment in the policy evaluation setting.
We describe the efficacy of these representations for policy evaluation, and
use our theoretical analysis to design new auxiliary learning rules. We
complement our theoretical results with an empirical comparison of these
learning rules for different cumulant functions on classic domains such as the
four-room domain (Sutton et al, 1999) and Mountain Car (Moore, 1990).

本文研究了强化学习中的状态表示问题，发现时序差分学习与蒙特卡罗、残差梯度学习在大部分环境的特征学习上存在差异，本文提出的新的辅助学习规则在经典环境下具有较好表现。