Lin Li
ABSTRACT
Adaptive traffic signal control is an important and challenging real-world problem that fits well with the task framework of deep reinforcement learning. As one of the critical design elements, the environmental state plays a crucial role in traffic signal control decisions. The state definitions of most existing works mostly contain lane-level queue length, intersection phase, and other features. However, these works are heuristically designed in representing states. This results in highly sensitive and unstable performances of next actions. The paper proposes a <u>C</u>ooperative <u>M</u>ax-<u>P</u>ressure enhanced <u>S</u>tate <u>L</u>earning for the traffic signal control (CMP-SL), which is inspired by the advanced pressure definition for an intersection in the transportation field to cope with this problem. First, our CMP-SL explicitly extends the cooperative max-pressure to the state definition of a target intersection, aiming to obtain accurate environment information by including the traffic pressures of surrounding intersections. From then on, a graph attention mechanism (GAT) is used to learn the state representation of the target intersection in our spatial-temporal state module. Second, since the state is coupled with the reward in reinforcement learning, our method takes the cooperative max-pressure of the target intersection into the reward definition. Furthermore, a temporal convolutional network (TCN) based sequence model is used to capture the historical state of traffic flow. And the historical spatial-temporal and the current spatial state features are concatenated into a DQN network to predict the Q value and generate each phase action. Finally, experiments with two real-world traffic datasets demonstrate that our method achieves shorter vehicle average times and higher network throughput than the state-of-the-art models.
- Jeancarlo Arguello Calvo and Ivana Dusparic. 2018. Heterogeneous Multi-Agent Deep Reinforcement Learning for Traffic Lights Control. In Proceedings for the 26th AIAI Irish Conference on Artificial Intelligence and Cognitive Science (AICS), Vol. 2259. 2--13.Google Scholar
- Chacha Chen, Hua Wei, Nan Xu, Guanjie Zheng, Ming Yang, Yuanhao Xiong, Kai Xu, and Zhenhui Li. 2020. Toward A Thousand Lights: Decentralized Deep Reinforcement Learning for Large-Scale Traffic Signal Control. In Proceedings of The Thirty-Fourth AAAI Conference on Artificial Intelligence. 3414--3421.Google ScholarCross Ref
- Tianshu Chu, Jie Wang, Lara Codecà, and Zhaojian Li. 2020. Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control. IEEE Transactions on Intelligent Transportation Systems, Vol. 21, 3 (2020), 1086--1095.Google ScholarCross Ref
- Mustafa Coskun, Abdelkader Baggag, and Sanjay Chawla. 2018. Deep Reinforcement Learning for Traffic Light Optimization. In 2018 IEEE International Conference on Data Mining Workshops (ICDMW). 564--571.Google Scholar
- Deepeka Garg, Maria Chli, and George Vogiatzis. 2018. Deep reinforcement learning for autonomous traffic light control. In 2018 3rd ieee international conference on intelligent transportation engineering (icite). IEEE, 214--218.Google Scholar
- Yaobang Gong, Mohamed Abdel-Aty, Qing Cai, and Md Sharikur Rahman. 2019. Decentralized network level adaptive signal control by multi-agent deep reinforcement learning. Transportation Research Interdisciplinary Perspectives, Vol. 1 (2019), 100020.Google ScholarCross Ref
- Shenxue Hao, Licai Yang, Li Ding, and Yajuan Guo. 2019. Distributed cooperative backpressure-based traffic light control method. Journal of Advanced Transportation, Vol. 2019 (2019), 1--15.Google Scholar
- Xiaoyuan Liang, Xusheng Du, Guiling Wang, and Zhu Han. 2019. A deep q learning network for traffic lights' cycle control in vehicular networks. IEEE Transactions on Vehicular Technology, Vol. 68, 2 (2019), 1243--1253.Google ScholarCross Ref
- Alan J Miller. 1963. Settings for fixed-cycle traffic signals. Journal of the Operational Research Society, Vol. 14, 4 (1963), 373--386.Google ScholarCross Ref
- Seyed Sajad Mousavi, Michael Schukat, Peter Corcoran, and Enda Howley. 2017. Traffic Light Control Using Deep Policy-Gradient and Value-Function Based Reinforcement Learning. IET Intelligent Transport Systems, Vol. 11, 7 (2017), 417--423.Google ScholarCross Ref
- Tomoki Nishi, Keisuke Otaki, Keiichiro Hayakawa, and Takayoshi Yoshimura. 2018. Traffic Signal Control Based on Reinforcement Learning with Graph Convolutional Neural Nets. In 21st International Conference on Intelligent Transportation Systems (ITSC). 877--883.Google Scholar
- Yuquan Peng, Lin Li, Qing Xie, and Xiaohui Tao. 2021. Learning Cooperative Max-Pressure Control by Leveraging Downstream Intersections Information for Traffic Signal Control. In Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint International Conference on Web and Big Data,, Leong Hou U, Marc Spaniol, Yasushi Sakurai, and Junying Chen (Eds.), Vol. 12859. 399--413.Google ScholarDigital Library
- Elise Van der Pol and Frans A Oliehoek. 2016. Coordinated deep reinforcement learners for traffic light control. Proceedings of learning, inference and control of multi-agent systems (at NIPS 2016), Vol. 1 (2016), 1--8.Google Scholar
- Pravin Varaiya. 2013. Max pressure control of a network of signalized intersections. Transportation Research Part C: Emerging Technologies, Vol. 36 (2013), 177--195.Google ScholarCross Ref
- Yanan Wang, Tong Xu, Xin Niu, Chang Tan, Enhong Chen, and Hui Xiong. 2020. STMARL: A spatio-temporal multi-agent reinforcement learning approach for cooperative traffic light control. IEEE Transactions on Mobile Computing (2020), 1--1.Google Scholar
- Hua Wei, Chacha Chen, Guanjie Zheng, Kan Wu, Vikash V. Gayah, Kai Xu, and Zhenhui Li. 2019a. PressLight: Learning Max Pressure Control to Coordinate Traffic Signals in Arterial Network. In In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1290--1298.Google ScholarDigital Library
- Hua Wei, Nan Xu, Huichu Zhang, Guanjie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yanmin Zhu, Kai Xu, and Zhenhui Li. 2019b. CoLight: Learning Network-level Cooperation for Traffic Signal Control. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM). 1913--1922.Google ScholarDigital Library
- Hua Wei, Guanjie Zheng, Vikash V. Gayah, and Zhenhui Li. 2020. Recent Advances in Reinforcement Learning for Traffic Signal Control: A Survey of Models and Evaluation. ACM SIGKDD Explorations Newsletter, Vol. 22, 2 (2020), 12--18.Google ScholarDigital Library
- Hua Wei, Guanjie Zheng, Huaxiu Yao, and Zhenhui Li. 2018. IntelliLight: A Reinforcement Learning Approach for Intelligent Traffic Light Control. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2496--2505.Google ScholarDigital Library
- Marco A Wiering. 2000. Multi-agent reinforcement learning for traffic light control. In Machine Learning: Proceedings of the Seventeenth International Conference (ICML). 1151--1158.Google Scholar
- Libing Wu, Min Wang, Dan Wu, and Jia Wu. 2021. DynSTGAT: Dynamic Spatial-Temporal Graph Attention Network for Traffic Signal Control. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (CIKM). 2150--2159.Google ScholarDigital Library
- Yuanhao Xiong, Guanjie Zheng, Kai Xu, and Zhenhui Li. 2019. Learning Traffic Signal Control from Demonstrations. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM). 2289--2292.Google ScholarDigital Library
- Xinshi Zang, Huaxiu Yao, Guanjie Zheng, Nan Xu, Kai Xu, and Zhenhui Li. 2020. MetaLight: Value-Based Meta-Reinforcement Learning for Traffic Signal Control. In Proceedings of The Thirty-Fourth AAAI Conference on Artificial Intelligence, Vol. 34. 1153--1160.Google ScholarCross Ref
- Ting Zhong, Zheyang Xu, and Fan Zhou. 2021. Probabilistic Graph Neural Networks for Traffic Signal Control. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 4085--4089.Google Scholar
Index Terms
- Cooperative Max-Pressure Enhanced Traffic Signal Control
Recommendations
Meta-Reinforcement Learning for Multiple Traffic Signals Control
CIKM '22: Proceedings of the 31st ACM International Conference on Information & Knowledge ManagementDespite the success of recent reinforcement learning (RL) in traffic signal control which has shown to outperform the conventional control methods, current RL-based methods require large amounts of samples to learn and lack the generalization ability to ...
Learning Cooperative Max-Pressure Control by Leveraging Downstream Intersections Information for Traffic Signal Control
Web and Big DataAbstractTraffic signal control problems are critical in urban intersections. Recently, deep reinforcement learning demonstrates impressive performance in the control of traffic signals. The design of state and reward function is often heuristic, which ...
Multi-mode Light: Learning Special Collaboration Patterns for Traffic Signal Control
Artificial Neural Networks and Machine Learning – ICANN 2022AbstractTo alleviate traffic congestion, it is a trend to apply reinforcement learning (RL) to traffic signal control in multi-intersection road networks. However, existing researches generally combine a basic RL framework Ape-X DQN with the graph ...
Comments