Deep Reinforcement Learning (DRL) policies are critically vulnerable to
adversarial noise in observations, posing severe risks in safety-critical
scenarios. For example, a self-driving car receiving manipulated sensory inputs
about traffic signs could lead to catastrophic outcomes. Existing strategies to
fortify RL algorithms against such adversarial perturbations generally fall
into two categories: (a) using regularization methods that enhance robustness
by incorporating adversarial loss terms into the value objectives, and (b)
adopting "maximin" principles, which focus on maximizing the minimum value to
ensure robustness. While regularization methods reduce the likelihood of
successful attacks, their effectiveness drops significantly if an attack does
succeed. On the other hand, maximin objectives, although robust, tend to be
overly conservative. To address this challenge, we introduce a novel objective
called Adversarial Counterfactual Error (ACoE), which naturally balances
optimizing value and robustness against adversarial attacks. To optimize ACoE
in a scalable manner in model-free settings, we propose a theoretically
justified surrogate objective known as Cumulative-ACoE (C-ACoE). The core idea
of optimizing C-ACoE is utilizing the belief about the underlying true state
given the adversarially perturbed observation. Our empirical evaluations
demonstrate that our method outperforms current state-of-the-art approaches for
addressing adversarial RL problems across all established benchmarks (MuJoCo,
Atari, and Highway) used in the literature.

深度强化学习中对抗性噪声的解决方法，包括使用正则化方法和引入 Adversarial Counterfactual Error 目标来提高鲁棒性。实证结果表明该方法在解决对抗 RL 问题上优于当前最先进的方法。