A Dynamic Cooperative Ramp Metering and Navigation Guidance Approach Based on Heterogeneous-Agent Reinforcement Learning

Ramp metering (RM) or navigation guidance (NG) plays a vital role in alleviating congestion. However, dynamically integrating RM and NG to achieve a “1+1>2” effect remains a challenge. To fill the technological gap, this article proposes a reinforcement learning-based dynamic cooperative RM and NG (DCRMNG) approach to conduct cooperative traffic control on the expressway system, which simultaneously adjusts demand and supply. Traffic congestion estimation (TCE) and route generation (RG) models are established for future traffic conditions prediction, traffic congestion evaluation, and guiding RG. To formulate coordinative control strategies for two distinct groups of agents, a heterogeneous-agent Markov decision process (HMDP) is developed. The reward function is carefully crafted to promote collaboration and accelerate the optimization algorithm’s convergence. Then, a novel heterogeneous-agent proximal policy optimization (HAPPO) algorithm is introduced to solve the DCRMNG approach. Finally, a real-world scenario with an expressway and its parallel arterial road in Hangzhou, China is simulated to assess the performance of the HAPPO-based DCRMNG approach. The results reveal that the proposed approach has the capability to improve the overall network traffic efficiency, mitigate the “navigation jam” issue and achieve intelligent and precise control of the expressway system, exhibiting outstanding performance in various traffic scenarios.