This study proposes a delay-compensated feedback controller based on proximal
policy optimization (PPO) reinforcement learning to stabilize traffic flow in
the congested regime by manipulating the time-gap of adaptive cruise
control-equipped (ACC-equipped) vehicles.The traffic dynamics on a freeway
segment are governed by an Aw-Rascle-Zhang (ARZ) model, consisting of $2\times
2$ nonlinear first-order partial differential equations (PDEs).Inspired by the
backstepping delay compensator [18] but different from whose complex segmented
control scheme, the PPO control is composed of three feedbacks, namely the
current traffic flow velocity, the current traffic flow density and previous
one step control input. The control gains for the three feedbacks are learned
from the interaction between the PPO and the numerical simulator of the traffic
system without knowing the system dynamics. Numerical simulation experiments
are designed to compare the Lyapunov control, the backstepping control and the
PPO control. The results show that for a delay-free system, the PPO control has
faster convergence rate and less control effort than the Lyapunov control. For
a traffic system with input delay, the performance of the PPO controller is
comparable to that of the Backstepping controller, even for the situation that
the delay value does not match. However, the PPO is robust to parameter
perturbations, while the Backstepping controller cannot stabilize a system
where one of the parameters is disturbed by Gaussian noise.

本研究提出了一种基于 PPO 强化学习的延迟补偿反馈控制器，通过控制自适应巡航控制（ACC）车辆的时间间隔来稳定拥堵区域的交通流量，使用 ARZ 模型对一段高速公路上的交通流进行仿真实验，结果表明，相对于 Lyapunov 控制，PPO 控制具有更快的收敛速度和更少的控制力，在具有输入延迟的交通系统中，PPO 控制器的性能与后方补偿控制器的性能相当。然而，PPO 对参数扰动具有鲁棒性，而后方补偿控制器无法稳定其中一个参数受高斯噪声扰动的系统。