Reinforcement Learning (RL) has recently achieved remarkable success in
robotic control. However, most RL methods operate in simulated environments
where privileged knowledge (e.g., dynamics, surroundings, terrains) is readily
available. Conversely, in real-world scenarios, robot agents usually rely
solely on local states (e.g., proprioceptive feedback of robot joints) to
select actions, leading to a significant sim-to-real gap. Existing methods
address this gap by either gradually reducing the reliance on privileged
knowledge or performing a two-stage policy imitation. However, we argue that
these methods are limited in their ability to fully leverage the privileged
knowledge, resulting in suboptimal performance. In this paper, we propose a
novel single-stage privileged knowledge distillation method called the
Historical Information Bottleneck (HIB) to narrow the sim-to-real gap. In
particular, HIB learns a privileged knowledge representation from historical
trajectories by capturing the underlying changeable dynamic information.
Theoretical analysis shows that the learned privileged knowledge representation
helps reduce the value discrepancy between the oracle and learned policies.
Empirical experiments on both simulated and real-world tasks demonstrate that
HIB yields improved generalizability compared to previous methods.

本文提出了一种称为历史信息瓶颈 (HIB) 的新型单阶段特权知识蒸馏方法，通过从历史轨迹中捕捉潜在的可变动态信息来学习特权知识表示，以弥合仿真与现实之间的差距，并证明该方法比现有方法具有更好的普适性。