While bisimulation-based approaches hold promise for learning robust state
representations for Reinforcement Learning (RL) tasks, their efficacy in
offline RL tasks has not been up to par. In some instances, their performance
has even significantly underperformed alternative methods. We aim to understand
why bisimulation methods succeed in online settings, but falter in offline
tasks. Our analysis reveals that missing transitions in the dataset are
particularly harmful to the bisimulation principle, leading to ineffective
estimation. We also shed light on the critical role of reward scaling in
bounding the scale of bisimulation measurements and of the value error they
induce. Based on these findings, we propose to apply the expectile operator for
representation learning to our offline RL setting, which helps to prevent
overfitting to incomplete data. Meanwhile, by introducing an appropriate reward
scaling strategy, we avoid the risk of feature collapse in representation
space. We implement these recommendations on two state-of-the-art
bisimulation-based algorithms, MICo and SimSR, and demonstrate performance
gains on two benchmark suites: D4RL and Visual D4RL. Codes are provided at
https://github.com/zanghyu/Offline_Bisimulation.

通过分析发现，离线强化学习任务中缺失的状态转换对于等价关系原则特别有害，导致估计无效。同时，奖励缩放在界定等价度量和其引发的值误差的尺度方面起着至关重要的作用。基于这些发现，我们提出在离线强化学习环境中应用期望分为运算符进行表示学习，从而防止对不完整数据进行过拟合。与此同时，通过引入适当的奖励缩放策略，我们避免了表示空间中特征崩溃的风险。我们在两个最先进的基于等价关系的算法 MICo 和 SimSR 上实现了这些建议，并在两个基准套件 D4RL 和 Visual D4RL 上展示了性能提升。代码可在 https://github.com/zanghyu/Offline_Bisimulation 中找到。