In this paper, we introduce a novel method for enhancing the effectiveness of
on-policy Deep Reinforcement Learning (DRL) algorithms. Current on-policy
algorithms, such as Proximal Policy Optimization (PPO) and Asynchronous
Advantage Actor-Critic (A3C), do not sufficiently account for cautious
interaction with the environment. Our method addresses this gap by explicitly
integrating cautious interaction in two critical ways: by maximizing a
lower-bound on the true value function plus a constant, thereby promoting a
\textit{conservative value estimation}, and by incorporating Thompson sampling
for cautious exploration. These features are realized through three
surprisingly simple modifications to the A3C algorithm: processing advantage
estimates through a ReLU function, spectral normalization, and dropout. We
provide theoretical proof that our algorithm maximizes the lower bound, which
also grounds Regret Matching Policy Gradients (RMPG), a discrete-action
on-policy method for multi-agent reinforcement learning. Our rigorous empirical
evaluations across various benchmarks consistently demonstrates our approach's
improved performance against existing on-policy algorithms. This research
represents a substantial step towards more cautious and effective DRL
algorithms, which has the potential to unlock application to complex,
real-world problems.

本文介绍了一种用于加强 on-policy 进行深度强化学习（DRL）算法的有效性的新方法，通过在价值函数估计中加入了保守性指标，同时使用 Thompson sampling 来实现谨慎探索，改进了现有算法，从而在多个基准中进行了严格的实证评估，并提供了理论证明，表明了新算法的可行性，特别是在多智能体强化学习中。