Deep reinforcement learning (DRL) algorithms can suffer from modeling errors
between the simulation and the real world. Many studies use adversarial
learning to generate perturbation during training process to model the
discrepancy and improve the robustness of DRL. However, most of these
approaches use a fixed parameter to control the intensity of the adversarial
perturbation, which can lead to a trade-off between average performance and
robustness. In fact, finding the optimal parameter of the perturbation is
challenging, as excessive perturbations may destabilize training and compromise
agent performance, while insufficient perturbations may not impart enough
information to enhance robustness. To keep the training stable while improving
robustness, we propose a simple but effective method, namely, Adaptive
Adversarial Perturbation (A2P), which can dynamically select appropriate
adversarial perturbations for each sample. Specifically, we propose an adaptive
adversarial coefficient framework to adjust the effect of the adversarial
perturbation during training. By designing a metric for the current intensity
of the perturbation, our method can calculate the suitable perturbation levels
based on the current relative performance. The appealing feature of our method
is that it is simple to deploy in real-world applications and does not require
accessing the simulator in advance. The experiments in MuJoCo show that our
method can improve the training stability and learn a robust policy when
migrated to different test environments. The code is available at
this https URL

深度强化学习算法中的模拟和真实世界之间的建模错误问题，通过使用对抗学习生成扰动以建模差异并提高深度强化学习的鲁棒性，在控制对抗性扰动的参数上存在平均性能和鲁棒性之间的权衡。为了保持训练的稳定性并提高鲁棒性，我们提出了一种简单而有效的方法，即自适应对抗性扰动 (A2P)，可以动态选择适当的对抗性扰动来处理每个样本。我们的方法具有将其部署在实际应用中且无需事先访问模拟器的优点。在 MuJoCo 上的实验表明，我们的方法可以提高训练的稳定性，并在不同的测试环境中学习到一种鲁棒的策略。