The excellent performance of Transformer in supervised learning has led to
growing interest in its potential application to deep reinforcement learning
(DRL) to achieve high performance on a wide variety of problems. However, the
decision making of a DRL agent is a black box, which greatly hinders the
application of the agent to real-world problems. To address this problem, we
propose the Action Q-Transformer (AQT), which introduces a transformer
encoder-decoder structure to Q-learning based DRL methods. In AQT, the encoder
calculates the state value function and the decoder calculates the advantage
function to promote the acquisition of different attentions indicating the
agent's decision-making. The decoder in AQT utilizes action queries, which
represent the information of each action, as queries. This enables us to obtain
the attentions for the state value and for each action. By acquiring and
visualizing these attentions that detail the agent's decision-making, we
achieve a DRL model with high interpretability. In this paper, we show that
visualization of attention in Atari 2600 games enables detailed analysis of
agents' decision-making in various game tasks. Further, experimental results
demonstrate that our method can achieve higher performance than the baseline in
some games.

本文提出了一种基于 Q-learning 的深度强化学习方法 ——Action Q-Transformer (AQT)，它引入了一个 Transformer 编码器 - 解码器结构来实现对代理决策的高度可解释性。我们利用 AQT 在 Atari 游戏任务中实现了代理的决策详细分析，并通过实验结果证明了该方法在某些游戏中可以实现比基线更高的性能。