In this work, we explore robust model-free reinforcement learning algorithms
for environments that may be dynamic or even adversarial. Conventional
state-based policies fail to accommodate the challenge imposed by the presence
of unmodeled disturbances in such settings. Additionally, optimizing linear
state-based policies pose obstacle for efficient optimization, leading to
nonconvex objectives even in benign environments like linear dynamical systems.
Drawing inspiration from recent advancements in model-based control, we
introduce a novel class of policies centered on disturbance signals. We define
several categories of these signals, referred to as pseudo-disturbances, and
corresponding policy classes based on them. We provide efficient and practical
algorithms for optimizing these policies.
Next, we examine the task of online adaptation of reinforcement learning
agents to adversarial disturbances. Our methods can be integrated with any
black-box model-free approach, resulting in provable regret guarantees if the
underlying dynamics is linear. We evaluate our method over different standard
RL benchmarks and demonstrate improved robustness.

该研究探讨了针对动态或敌对环境的强化学习算法，并引入基于干扰信号的新型政策以提高适应性和鲁棒性，并在标准 RL 基准测试中进行了评估并证明了改进的鲁棒性。