Decision Transformers, in their vanilla form, struggle to perform on image-based environments with multi-discrete action spaces. Although enhanced Decision Transformer architectures have been developed to improve performance, these methods have not specifically addressed this problem of multi-discrete action spaces which hampers existing Decision Transformer architectures from learning good representations. To mitigate this, we propose Multi-State Action Tokenisation (M-SAT), an approach for tokenising actions in multi-discrete action spaces that enhances the model's performance in such environments. Our approach involves two key changes: disentangling actions to the individual action level and tokenising the actions with auxiliary state information. These two key changes also improve individual action level interpretability and visibility within the attention layers. We demonstrate the performance gains of M-SAT on challenging ViZDoom environments with multi-discrete action spaces and image-based state spaces, including the Deadly Corridor and My Way Home scenarios, where M-SAT outperforms the baseline Decision Transformer without any additional data or heavy computational overheads. Additionally, we find that removing positional encoding does not adversely affect M-SAT's performance and, in some cases, even improves it.

我们提出了Multi-State Action Tokenisation (M-SAT)方法，针对具有多离散行动空间的图像环境对决策变压器模型的性能进行改进，通过将行动进行细分并结合辅助状态信息进行标记化，实现了在这些环境中模型性能的提升。在具有多离散行动空间和基于图像状态空间的ViZDoom环境中进行的实验表明，M-SAT在没有额外数据或大量计算开销的情况下优于基准的决策变压器模型，并且我们发现去除位置编码对M-SAT的性能没有负面影响，并在某些情况下甚至会提升性能。

决策变压器中的多状态动作令牌化在多离散动作空间中的应用