Robust reinforcement learning (RL) seeks to train policies that can perform
well under environment perturbations or adversarial attacks. Existing
approaches typically assume that the space of possible perturbations remains
the same across timesteps. However, in many settings, the space of possible
perturbations at a given timestep depends on past perturbations. We formally
introduce temporally-coupled perturbations, presenting a novel challenge for
existing robust RL methods. To tackle this challenge, we propose GRAD, a novel
game-theoretic approach that treats the temporally-coupled robust RL problem as
a partially-observable two-player zero-sum game. By finding an approximate
equilibrium in this game, GRAD ensures the agent's robustness against
temporally-coupled perturbations. Empirical experiments on a variety of
continuous control tasks demonstrate that our proposed approach exhibits
significant robustness advantages compared to baselines against both standard
and temporally-coupled attacks, in both state and action spaces.

提出了一个新的挑战，即通过 GRAD 方法将强化学习中的时间上耦合的扰动问题视为部分可观察的两人零和游戏，并通过在该游戏中找到逼近均衡来确保智能体对于时间上耦合的扰动的鲁棒性，实验结果表明，相比于基准方法，在状态空间和动作空间中，所提出的方法在多种连续控制任务上对抗标准攻击和时间上耦合攻击都展现出显著的鲁棒性优势。