In this paper, we propose the Phy-DRL: a physics-model-regulated deep
reinforcement learning framework for safety-critical autonomous systems. The
Phy-DRL is unique in three innovations: i) proactive unknown-unknowns training,
ii) conjunctive residual control (i.e., integration of data-driven control and
physics-model-based control) and safety- \& stability-sensitive reward, and
iii) physics-model-based neural network editing, including link editing and
activation editing. Thanks to the concurrent designs, the Phy-DRL is able to 1)
tolerate unknown-unknowns disturbances, 2) guarantee mathematically provable
safety and stability, and 3) strictly comply with physical knowledge pertaining
to Bellman equation and reward. The effectiveness of the Phy-DRL is finally
validated by an inverted pendulum and a quadruped robot. The experimental
results demonstrate that compared with purely data-driven DRL, Phy-DRL features
remarkably fewer learning parameters, accelerated training and enlarged reward,
while offering enhanced model robustness and safety assurance.

本文中，我们提出了 Phy-DRL 框架：一种物理模型调节的深度强化学习框架，用于安全关键的自主系统。Phy-DRL 在三个方面独具创新：积极的未知 - 未知训练，连接残余控制，以及基于物理模型的神经网络编辑。在倒立摆和四足机器人上进行的实验结果证明了与纯数据驱动 DRL 相比，Phy-DRL 具有更少的学习参数，卓越的模型鲁棒性和安全保证。