Representation learning and exploration are among the key challenges for any
deep reinforcement learning agent. In this work, we provide a singular value
decomposition based method that can be used to obtain representations that
preserve the underlying transition structure in the domain. Perhaps
interestingly, we show that these representations also capture the relative
frequency of state visitations, thereby providing an estimate for pseudo-counts
for free. To scale this decomposition method to large-scale domains, we provide
an algorithm that never requires building the transition matrix, can make use
of deep networks, and also permits mini-batch training. Further, we draw
inspiration from predictive state representations and extend our decomposition
method to partially observable environments. With experiments on multi-task
settings with partially observable domains, we show that the proposed method
can not only learn useful representation on DM-Lab-30 environments (that have
inputs involving language instructions, pixel images, and rewards, among
others) but it can also be effective at hard exploration tasks in DM-Hard-8
environments.

该研究论文介绍了一个基于奇异值分解的方法，用于在领域中保留基础转换结构的表示，从而提供伪计数的估计，在多任务中展示了结果，并解决了部分可观测的环境下的难以探索的任务。

使用奇异值分解的深度强化学习表示与探索

Representations and Exploration for Deep Reinforcement Learning using  Singular Value Decomposition

Although exploration in reinforcement learning is well understood from a
theoretical point of view, provably correct methods remain impractical. In this
paper we study the interplay between exploration and approximation, what we
call approximate exploration. Our main goal is to further our theoretical
understanding of pseudo-count based exploration bonuses (Bellemare et al.,
2016), a practical exploration scheme based on density modelling. As a warm-up,
we quantify the performance of an exploration algorithm, MBIE-EB (Strehl and
Littman, 2008), when explicitly combined with state aggregation. This allows us
to confirm that, as might be expected, approximation allows the agent to trade
off between learning speed and quality of the learned policy. Next, we show how
a given density model can be related to an abstraction and that the
corresponding pseudo-count bonus can act as a substitute in MBIE-EB combined
with this abstraction, but may lead to either under- or over-exploration. Then,
we show that a given density model also defines an implicit abstraction, and
find a surprising mismatch between pseudo-counts derived either implicitly or
explicitly. Finally we derive a new pseudo-count bonus alleviating this issue.

研究强化学习中探索和近似之间的相互作用，提出一种基于密度建模的方法来改善探索，探讨伪计数奖励在此方法中的应用，发现了在其应用中可能存在的过度或不足探索问题，并提出一种新的伪计数奖励来缓解这些问题。