We investigate the impact of auxiliary learning tasks such as observation
reconstruction and latent self-prediction on the representation learning
problem in reinforcement learning. We also study how they interact with
distractions and observation functions in the MDP. We provide a theoretical
analysis of the learning dynamics of observation reconstruction, latent
self-prediction, and TD learning in the presence of distractions and
observation functions under linear model assumptions. With this formalization,
we are able to explain why latent-self prediction is a helpful \emph{auxiliary
task}, while observation reconstruction can provide more useful features when
used in isolation. Our empirical analysis shows that the insights obtained from
our learning dynamics framework predicts the behavior of these loss functions
beyond the linear model assumption in non-linear neural networks. This
reinforces the usefulness of the linear model framework not only for
theoretical analysis, but also practical benefit for applied problems.

我们研究了辅助学习任务对强化学习中的表示学习问题的影响，包括观测重建和潜在自预测，并研究它们如何与干扰项和观测函数在 MDP 中交互。

自我预测何时有帮助？理解增强学习中的辅助任务

When does Self-Prediction help? Understanding Auxiliary Tasks in  Reinforcement Learning

ObjectGoal Navigation (ObjectNav) is an embodied task wherein agents are to
navigate to an object instance in an unseen environment. Prior works have shown
that end-to-end ObjectNav agents that use vanilla visual and recurrent modules,
e.g. a CNN+RNN, perform poorly due to overfitting and sample inefficiency. This
has motivated current state-of-the-art methods to mix analytic and learned
components and operate on explicit spatial maps of the environment. We instead
re-enable a generic learned agent by adding auxiliary learning tasks and an
exploration reward. Our agents achieve 24.5% success and 8.1% SPL, a 37% and 8%
relative improvement over prior state-of-the-art, respectively, on the Habitat
ObjectNav Challenge. From our analysis, we propose that agents will act to
simplify their visual inputs so as to smooth their RNN dynamics, and that
auxiliary tasks reduce overfitting by minimizing effective RNN dimensionality;
i.e. a performant ObjectNav agent that must maintain coherent plans over long
horizons does so by learning smooth, low-dimensional recurrent dynamics. Site:
this https URL

通过添加辅助学习任务和探索奖励，本文重新启用了一个通用的学习代理，从而实现了对 ObjectNav 挑战的目标导航。分析表明，代理将简化其视觉输入，以平滑其循环神经网络动态，并且辅助任务减少过度拟合。