Decision Transformer (DT), which employs expressive sequence modeling
techniques to perform action generation, has emerged as a promising approach to
offline policy optimization. However, DT generates actions conditioned on a
desired future return, which is known to bear some weaknesses such as the
susceptibility to environmental stochasticity. To overcome DT's weaknesses, we
propose to empower DT with dynamic programming. Our method comprises three
steps. First, we employ in-sample value iteration to obtain approximated value
functions, which involves dynamic programming over the MDP structure. Second,
we evaluate action quality in context with estimated advantages. We introduce
two types of advantage estimators, IAE and GAE, which are suitable for
different tasks. Third, we train an Advantage-Conditioned Transformer (ACT) to
generate actions conditioned on the estimated advantages. Finally, during
testing, ACT generates actions conditioned on a desired advantage. Our
evaluation results validate that, by leveraging the power of dynamic
programming, ACT demonstrates effective trajectory stitching and robust action
generation in spite of the environmental stochasticity, outperforming baseline
methods across various benchmarks. Additionally, we conduct an in-depth
analysis of ACT's various design choices through ablation studies.

使用决策转换器（DT）和动态规划相结合的方法（ACT），克服了动态规划的弱点，通过有效的轨迹拼接和鲁棒的动作生成，在环境随机性方面表现出色，优于各种基准方法。

ACT：通过优势调节实现决策变换的动态规划赋能

ACT: Empowering Decision Transformer with Dynamic Programming via  Advantage Conditioning

We propose to decompose instruction execution to goal prediction and action
generation. We design a model that maps raw visual observations to goals using
LINGUNET, a language-conditioned image generation network, and then generates
the actions required to complete them. Our model is trained from demonstration
only without external resources. To evaluate our approach, we introduce two
benchmarks for instruction following: LANI, a navigation task; and CHAI, where
an agent executes household instructions. Our evaluation demonstrates the
advantages of our model decomposition, and illustrates the challenges posed by
our new benchmarks.

该研究倡导将指令执行分解为目标预测和行为生成，并介绍了一种利用 LINGUNET 将观察值映射到目标，然后生成完成目标所需的行动的模型。该模型仅通过演示进行训练，并引入了两个指令跟随基准来评估该方法：LANI 和 CHAI。我们的评估证明了我们的模型分解的优势，并阐明了我们的新基准所提出的挑战。