In Reinforcement Learning (RL), Laplacian Representation (LapRep) is a
task-agnostic state representation that encodes the geometry of the
environment. A desirable property of LapRep stated in prior works is that the
Euclidean distance in the LapRep space roughly reflects the reachability
between states, which motivates the usage of this distance for reward shaping.
However, we find that LapRep does not necessarily have this property in
general: two states having small distance under LapRep can actually be far away
in the environment. Such mismatch would impede the learning process in reward
shaping. To fix this issue, we introduce a Reachability-Aware Laplacian
Representation (RA-LapRep), by properly scaling each dimension of LapRep.
Despite the simplicity, we demonstrate that RA-LapRep can better capture the
inter-state reachability as compared to LapRep, through both theoretical
explanations and experimental results. Additionally, we show that this
improvement yields a significant boost in reward shaping performance and also
benefits bottleneck state discovery.

该研究论文介绍了一种基于 Laplacian Representation 并能够有效解决 inter-state reachability 问题的任务无关状态表示方法 RA-LapRep, 该方法能够显著提高 reward shaping 的性能以及帮助发现 bottleneck 状态。