Reinforcement learning (RL) is inherently rife with non-stationarity since
the states and rewards the agent observes during training depend on its
changing policy. Therefore, networks in deep RL must be capable of adapting to
new observations and fitting new targets. However, previous works have observed
that networks in off-policy deep value-based methods exhibit a decrease in
representation rank, often correlated with an inability to continue learning or
a collapse in performance. Although this phenomenon has generally been
attributed to neural network learning under non-stationarity, it has been
overlooked in on-policy policy optimization methods which are often thought
capable of training indefinitely. In this work, we empirically study
representation dynamics in Proximal Policy Optimization (PPO) on the Atari and
MuJoCo environments, revealing that PPO agents are also affected by feature
rank deterioration and loss of plasticity. We show that this is aggravated with
stronger non-stationarity, ultimately driving the actor's performance to
collapse, regardless of the performance of the critic. We draw connections
between representation collapse, performance collapse, and trust region issues
in PPO, and present Proximal Feature Optimization (PFO), a novel auxiliary
loss, that along with other interventions shows that regularizing the
representation dynamics improves the performance of PPO agents.

通过对 Atari 和 MuJoCo 环境中的 PPO 代理的实证研究，揭示了 PPO 代理受特征秩降低和可塑性丧失的影响，这一现象加剧了强非稳态性，最终导致演员的性能崩溃，无论评论家的性能如何。我们建立了表示崩溃、性能崩溃和 PPO 中的信任域问题之间的联系，并提出了一种名为 PFO 的新型辅助损失，通过调节表示动态改善 PPO 代理的性能。