Despite the recent advancements in offline reinforcement learning via supervised learning (RvS) and the success of the decision transformer (DT) architecture in various domains, DTs have fallen short in several challenging benchmarks. The root cause of this underperformance lies in their inability to seamlessly connect segments of suboptimal trajectories. To overcome this limitation, we present a novel approach to enhance RvS methods by integrating intermediate targets. We introduce the Waypoint Transformer (WT), using an architecture that builds upon the DT framework and conditioned on automatically-generated waypoints. The results show a significant increase in the final return compared to existing RvS methods, with performance on par or greater than existing state-of-the-art temporal difference learning-based methods. Additionally, the performance and stability improvements are largest in the most challenging environments and data configurations, including AntMaze Large Play/Diverse and Kitchen Mixed/Partial.

本研究提出了一种增强离线强化学习方法的新方法，即引入中间节点的 Waypoint Transformer，建立在决策变压器 (Decision Transformer) 框架上，自动产生的中间节点有助于连接不完美的轨迹，从而提高离线强化学习的性能和稳定性，并在各种负责的环境和数据配置中表现出与现有最先进的基于时间差异的学习方法相当或更优的性能。

Waypoint Transformer：通过中间目标的监督学习进行强化学习