Highway merging scenarios featuring mixed traffic conditions pose significant
modeling and control challenges for connected and automated vehicles (CAVs)
interacting with incoming on-ramp human-driven vehicles (HDVs). In this paper,
we present an approach to learn an approximate information state model of
CAV-HDV interactions for a CAV to maneuver safely during highway merging. In
our approach, the CAV learns the behavior of an incoming HDV using approximate
information states before generating a control strategy to facilitate merging.
First, we validate the efficacy of this framework on real-world data by using
it to predict the behavior of an HDV in mixed traffic situations extracted from
the Next-Generation Simulation repository. Then, we generate simulation data
for HDV-CAV interactions in a highway merging scenario using a standard inverse
reinforcement learning approach. Without assuming a prior knowledge of the
generating model, we show that our approximate information state model learns
to predict the future trajectory of the HDV using only observations.
Subsequently, we generate safe control policies for a CAV while merging with
HDVs, demonstrating a spectrum of driving behaviors, from aggressive to
conservative. We demonstrate the effectiveness of the proposed approach by
performing numerical simulations.

该论文提出了一种学习连接和自动化车辆 (CAV) 与进入匝道的人驾驶车辆 (HDV) 进行安全控制的方法，使用高度信息状态模型，学习了混合交通情况下 HDV 的行为，并生成安全控制策略，然后在 CAV 与 HDV 交互的模拟场景中进行了验证。

混合交通中的联网自动驾驶车辆：学习人类驾驶员行为以实现有效的匝道并入

Connected and Automated Vehicles in Mixed-Traffic: Learning Human Driver  Behavior for Effective On-Ramp Merging

On-ramp merging is a challenging task for autonomous vehicles (AVs),
especially in mixed traffic where AVs coexist with human-driven vehicles
(HDVs). In this paper, we formulate the mixed-traffic highway on-ramp merging
problem as a multi-agent reinforcement learning (MARL) problem, where the AVs
(on both merge lane and through lane) collaboratively learn a policy to adapt
to HDVs to maximize the traffic throughput. We develop an efficient and
scalable MARL framework that can be used in dynamic traffic where the
communication topology could be time-varying. Parameter sharing and local
rewards are exploited to foster inter-agent cooperation while achieving great
scalability. An action masking scheme is employed to improve learning
efficiency by filtering out invalid/unsafe actions at each step. In addition, a
novel priority-based safety supervisor is developed to significantly reduce
collision rate and greatly expedite the training process. A gym-like simulation
environment is developed and open-sourced with three different levels of
traffic densities. We exploit curriculum learning to efficiently learn harder
tasks from trained models under simpler settings. Comprehensive experimental
results show the proposed MARL framework consistently outperforms several
state-of-the-art benchmarks.

本文通过多智能体强化学习的方法对自动驾驶车辆在复杂交通场景下的高速公路入口合流进行了建模，提出了一种高效可扩展的模型，包括参数共享、本地奖励、底层安全监督等手段以及基于优先级的安全监督等，最终实验结果表明本文提出的 MARL 框架明显优于现有的基准模型。