As a data-driven paradigm, offline reinforcement learning (RL) has been
formulated as sequence modeling that conditions on the hindsight information
including returns, goal or future trajectory. Although promising, this
supervised paradigm overlooks the core objective of RL that maximizes the
return. This overlook directly leads to the lack of trajectory stitching
capability that affects the sequence model learning from sub-optimal data. In
this work, we introduce the concept of max-return sequence modeling which
integrates the goal of maximizing returns into existing sequence models. We
propose Reinforced Transformer (Reinformer), indicating the sequence model is
reinforced by the RL objective. Reinformer additionally incorporates the
objective of maximizing returns in the training phase, aiming to predict the
maximum future return within the distribution. During inference, this
in-distribution maximum return will guide the selection of optimal actions.
Empirically, Reinformer is competitive with classical RL methods on the D4RL
benchmark and outperforms state-of-the-art sequence model particularly in
trajectory stitching ability. Code is public at
https://github.com/Dragon-Zhuang/Reinformer.

该论文介绍了最大回报的序列建模概念，提出了 Reinforced Transformer（Reinformer）作为一个整合了最大化回报目标的序列模型。Reinformer 在训练阶段将最大化回报的目标纳入模型，并在推断时引导最优动作的选择，从而在轨迹拼接能力上优于传统强化学习方法和现有的序列模型。

离线强化学习的最大回报序列建模

Reinformer: Max-Return Sequence Modeling for offline RL

This paper introduces Elastic Decision Transformer (EDT), a significant
advancement over the existing Decision Transformer (DT) and its variants.
Although DT purports to generate an optimal trajectory, empirical evidence
suggests it struggles with trajectory stitching, a process involving the
generation of an optimal or near-optimal trajectory from the best parts of a
set of sub-optimal trajectories. The proposed EDT differentiates itself by
facilitating trajectory stitching during action inference at test time,
achieved by adjusting the history length maintained in DT. Further, the EDT
optimizes the trajectory by retaining a longer history when the previous
trajectory is optimal and a shorter one when it is sub-optimal, enabling it to
"stitch" with a more optimal trajectory. Extensive experimentation demonstrates
EDT's ability to bridge the performance gap between DT-based and Q
Learning-based approaches. In particular, the EDT outperforms Q Learning-based
methods in a multi-task regime on the D4RL locomotion benchmark and Atari
games. Videos are available at: this https URL

本研究介绍了一种新的弹性决策变换器（EDT）方法，可以在测试时通过调整在 DT 中保留的历史长度来促进轨迹拼接，实现优化轨迹和跨越 DT 和 Q 学习方法之间的性能差距，表现出优越性能。