Recent work has shown that offline reinforcement learning (RL) can be
formulated as a sequence modeling problem (Chen et al., 2021; Janner et al.,
2021) and solved via approaches similar to large-scale language modeling.
However, any practical instantiation of RL also involves an online component,
where policies pretrained on passive offline datasets are finetuned via
taskspecific interactions with the environment. We propose Online Decision
Transformers (ODT), an RL algorithm based on sequence modeling that blends
offline pretraining with online finetuning in a unified framework. Our
framework uses sequence-level entropy regularizers in conjunction with
autoregressive modeling objectives for sample-efficient exploration and
finetuning. Empirically, we show that ODT is competitive with the
state-of-the-art in absolute performance on the D4RL benchmark but shows much
more significant gains during the finetuning procedure.

本文提出了基于序列建模的决策转换器（ODT）算法，该算法在离线预训练和在线调整中融合了序列级熵正则化和自回归建模目标，以实现高效的探索和调整。实验证明，在 D4RL 基准测试中，ODT 在绝对性能方面与最先进的方法具有竞争力，在微调过程中展现出更显著的提高。