This study proposes a framework for human-like autonomous car-following
planning based on deep reinforcement learning (deep RL). Historical driving
data are fed into a simulation environment where an RL agent learns from trial
and error interactions based on a reward function that signals how much the
agent deviates from the empirical data. Through these interactions, an optimal
policy, or car-following model that maps in a human-like way from speed,
relative speed between a lead and following vehicle, and inter-vehicle spacing
to acceleration of a following vehicle is finally obtained. The model can be
continuously updated when more data are fed in. Two thousand car-following
periods extracted from the 2015 Shanghai Naturalistic Driving Study were used
to train the model and compare its performance with that of traditional and
recent data-driven car-following models. As shown by this study results, a deep
deterministic policy gradient car-following model that uses disparity between
simulated and observed speed as the reward function and considers a reaction
delay of 1s, denoted as DDPGvRT, can reproduce human-like car-following
behavior with higher accuracy than traditional and recent data-driven
car-following models. Specifically, the DDPGvRT model has a spacing validation
error of 18% and speed validation error of 5%, which are less than those of
other models, including the intelligent driver model, models based on locally
weighted regression, and conventional neural network-based models. Moreover,
the DDPGvRT demonstrates good capability of generalization to various driving
situations and can adapt to different drivers by continuously learning. This
study demonstrates that reinforcement learning methodology can offer insight
into driver behavior and can contribute to the development of human-like
autonomous driving algorithms and traffic-flow models.

该研究提出了一种基于深度强化学习的类人自动车跟随规划框架，并通过创新的奖励函数和反应延迟考虑等方法，将人的驾驶行为映射到速度、相对速度和车辆间距离等状态空间，实现了精度更高的自动驾驶策略学习，具有较强泛化能力，可以为智能驾驶算法和交通流模型的发展提供帮助。