Deep reinforcement learning (DRL) is becoming increasingly popular in
implementing traffic signal control (TSC). However, most existing DRL methods
employ fixed control strategies, making traffic signal phase duration less
flexible. Additionally, the trend of using more complex DRL models makes
real-life deployment more challenging. To address these two challenges, we
firstly propose a two-stage DRL framework, named DynamicLight, which uses Max
Queue-Length to select the proper phase and employs a deep Q-learning network
to determine the duration of the corresponding phase. Based on the design of
DynamicLight, we also introduce two variants: (1) DynamicLight-Lite, which
addresses the first challenge by using only 19 parameters to achieve dynamic
phase duration settings; and (2) DynamicLight-Cycle, which tackles the second
challenge by actuating a set of phases in a fixed cyclical order to implement
flexible phase duration in the respective cyclical phase structure. Numerical
experiments are conducted using both real-world and synthetic datasets,
covering four most commonly adopted traffic signal intersections in real life.
Experimental results show that: (1) DynamicLight can learn satisfactorily on
determining the phase duration and achieve a new state-of-the-art, with
improvement up to 6% compared to the baselines in terms of adjusted average
travel time; (2) DynamicLight-Lite matches or outperforms most baseline methods
with only 19 parameters; and (3) DynamicLight-Cycle demonstrates high
performance for current TSC systems without remarkable modification in an
actual deployment. Our code is released at Github.

本文提出了一个名为 DynamicLight 的两阶段深度强化学习框架，它使用最大队列长度选择正确阶段，并采用深度 Q 学习网络确定相应阶段的持续时间。在此基础上，还引入了两个变体。数字实验表明，DynamicLight 在确定相位持续时间方面可以满足，DynamicLight-Lite 可以与大多数基线方法相匹配或优于大部分基线方法，DynamicLight-Cycle 在当前 TSC 系统方面表现出高性能。