Academic Journal of Science and Technology ISSN: 2771-3032 | Vol. 6, No. 1, 2023 215 Research on the Control Method of Urban Expressway Weaving Area Huayu Xia College of Traffic & Transportation, Chongqing Jiaotong University, Chongqing, 400074, China Abstract: With the rapid development of urban expressway construction, there are some problems in the actual operation process, which leads to the traffic efficiency not reaching the expected level. Due to the long-term and large amount of actual data is difficult to obtain, this paper uses the method of combining actual data and simulation to explore the traffic operation law of urban expressway weaving area, analyzes the factors affecting the traffic efficiency of weaving area, and applies the variable speed limit guidance control method based on DQN algorithm, which can effectively improve the traffic efficiency of urban expressway weaving area. Keywords: Urban expressway weaving area, Traffic efficiency, Deep reinforcement learning, Vissim-python simulation. 1. Introduction With the continuous increase of urban expressway construction mileage, people's travel needs are also constantly increasing. During peak travel hours, traffic congestion is becoming increasingly severe, and it can also cause traffic safety issues. The construction of urban expressways has greatly improved travel efficiency. However, compared to highways, they have the characteristics of short ramp spacing and multiple nodes. During peak travel hours, they have tidal characteristics and are prone to traffic congestion at the bottleneck of weaving sections, especially for weaving areas with close access to and exit ramps. Traffic congestion is more severe and does not play a good role in rapid evacuation in the backbone system. The complex interweaving phenomenon of traffic flow within the bottleneck sections of urban expressways has a significant impact on the capacity of the main line, making it difficult for vehicles upstream of the main line to enter the exit ramp, and for vehicles on the entrance ramp to enter the main line, resulting in much lower traffic efficiency than expected. The vast majority of urban expressways in China are transformed from expressways. Before being transformed into expressways, there were many merging and diverging points on the main line, and the spacing between entrances and exits was small, resulting in a small length of weaving zones [1]. Due to various factors and conditions, many expressways would be difficult to meet the design specifications if they were transformed according to existing standards. When the length of the weaving section is too short, it not only causes congestion problems, but also safety issues. At the same time, the methods to solve congestion in urban expressway weaving areas are not only considered from a design perspective, but also commonly used control methods, such as channel control, ramp control, variable speed limit guidance control, etc. Due to the large input flow of the main line of urban expressways, the phenomenon of weaving is severe, which is easily affected by the internal traffic flow of the weaving area and leads to the propagation of vehicles upstream of the main line. At this time, the evacuation capacity of vehicles inside the weaving area is lower than the capacity of the road section, resulting in a "capacity drop" phenomenon [2]. Therefore, effectively preventing the occurrence of the phenomenon is the key to ensuring smooth and efficient operation of traffic flow on the main line. According to practical application results, variable speed limit guidance control (VSL) can reduce the dispersion of traffic flow speed, thereby reducing the occurrence of traffic safety accidents. On the other hand, it makes the traffic flow speed more balanced, and to some extent, improves the traffic efficiency of the weaving area. 2. Related Work This article selects an expressway in Chongqing as the research object for the Entry-Exit type weaving area. In order to ensure the accuracy of data collection, we used a combination of on-site video cameras and manual counting to collect traffic data. The content of data collection includes the following two aspects: first, the geometric conditions of the actual road; The second is traffic flow data and weaving zone performance indicators. The collected data, after unified processing and calculation, is shown in Figure 1, Table 1, and Table 2. Figure 1. Actual Geometry Conditions of the Road Table 1. Geometry Conditions Geometric Conditions of Roads Design Speed (km/h) LaneNumber Width of Lane (m) Upstream 80 4 3.75 Downstream 80 4 3.75 On-ramp 35 2 3.5 Exit-ramp 35 2 3.5 216 Table 2. Flow Data and Average Travel Time in Weaving Zones Time Upstream Downstream On- ramp Exit- ramp Avg- speed 7:30- 8:30 Car (4236) Car(4031) Car (869) Car (1469) 34.6 Bus (54) Bus(53) Bus (29) Bus (29) 17:30- 18:30 Car (4816) Car(4169) Car (642) Car (1465) 29.2 Bus (62) Bus(59) Bus (16) Bus (28) Using the geometric conditions of the road surveyed on site and the peak traffic flow data in the morning and evening, a VISSIM traffic simulation model was established. When establishing the model, path decisions were set for the upstream input flow of the main line and the input flow of the on-ramp, respectively. At the same time, priority traffic rules were set up in the merging area of the main line and the ramp. When there are gaps on the main line, the traffic flow of the on-ramp can only merge; When the traffic flow is about to reach the merging or diverging area, vehicles are often affected by the interweaving of traffic flow to slow down, so a deceleration zone needs to be set up; Due to the different expected speeds of vehicles entering the mainline and ramps, acceleration and deceleration are required when entering the mainline and exiting the exit ramp. Therefore, expected speed decision points are set at the entrance and exit ramps, and the established model is as follows. Figure 2. VISSIM Traffic Simulation Model 3. Variable Speed Limit Control Based on DQN 3.1. DQN Settings The key elements of variable speed limit guidance control based on DQN algorithm are designed as follows: (1) State Definition. By setting data collection points in the variable speed limit guidance control area upstream of the main line, weaving section, downstream section of the main line, and at the entrance of the on-ramp, the real-time traffic flow and speed values can be obtained through calculation and conversion. Then, based on the number of lanes, the average density value of a single lane can be further obtained, and the corresponding service level can be determined by the average density value of a single lane, Finally, real-time service levels are used as input to the state space, and DQN adopts greedy algorithm to explore the state space. (2) Action Definition. Select the variable speed limit value as the action, and based on relevant standards and regulations on the variable speed limit value according to the driver's compliance rate, while considering the driver's comfort and safety, set the variable speed limit value as 30km/h, 40km/h, 50km/h, 60km/h, 70km/h, 80km/h, that is, there is an action set, which includes actions that do not adopt speed limits. This means that the system will independently learn and determine whether to take speed limit measures [3]. (3) Reward Definition. The purpose of the expressway control system is to improve the traffic efficiency of the weaving area. In the road network, the average delay and number of stops of vehicles can effectively reflect the overall traffic efficiency of the road network, and it is also convenient to evaluate the actual control effect [4]. Due to the use of maximum calculation in Q value updates, the goal of the control system is to improve the traffic efficiency of the road network, which is to reduce the average delay and number of stops of vehicles in the road network, Therefore, the reward function of the control system in this paper uses the opposite number of the sum of the average delay of the road network vehicles and the number of stops as the reward function. The formula is defined as Rt = -(D(t) + P(t)). (1) Where D(t), P(t) Represent the average delay and number of stops of vehicles on the road network during the t-th control cycle, respectively. (4) The training process of DQN. Before learning and training, it is necessary to first construct a neural network and initialize parameters. This article uses a structural network in the TensorFlow framework, which initializes parameters such as learning rate (LR), reduction factor 𝛾, decay rate 𝛼, initial 𝜀, minimum 𝜀, size of experience playback memory, and size of extracted samples. Based on the preliminary testing results, the specific parameters are set as follows: learning rate (LR) of 0.01, reduction factor 𝛾 of 0.95, attenuation rate 𝛼 of 0.999, initial value 𝜀 of 1, minimum value 𝜀 of 0.1, size of experience playback memory library of 10000, number of extracted samples of 8. After initializing the parameters, simulate and run a control cycle without taking variable speed limit measures, The initial traffic data within the variable speed limit guidance control area can be obtained through data collection points; Then, before starting the simulation for each control cycle, based on the size of 𝜀, a variable speed limit value is randomly selected in the action set with a certain probability [5]. Then, the Q value is selected to update the maximum variable speed limit value. The obtained speed limit value is used as the desired speed decision point setting speed through Python calling the VISSIM-COM interface, and the simulation is continued for one control cycle. During this period, the traffic status, variable speed limit value Store the reward and the traffic status at the next moment in the experience playback memory library; Continuously cycle the above steps until the set observation time is completed; After the observation time is over, the network parameters are continuously adjusted according to the Q-value function, while the values are continuously reduced, so that the neural network can tend to have the sum of average vehicle delay and parking times in the road network near the minimum state for a long time. 217 3.2. Acquisition and Processing of Vehicle Trajectory Data (1) Obtain trajectory data. In addition to analyzing the smoothness of road network traffic from road network evaluation indicators, the operating status of vehicle flow can also be intuitively observed from the driving trajectory of vehicles in the road network, and the smoothness of road network traffic can be analyzed. The author selects the vehicle trajectory setting in the VISSIM configuration, and after the simulation run is completed, a file will be output in the file. The data content in the file includes the simulation time and vehicle number, section number, lane number, vehicle position, and vehicle lateral position [6]. (2) Processing methods for trajectory data. According to the composition of the traffic simulation model, the road sections can be divided into main line sections, on ramp sections, off ramp sections, and connecting sections. Each section has a corresponding number in the model, where the main line section includes section number 1, section number 2, and section number 3; The entrance ramp section is section number 4; The off-ramp section is section number 5, and the connecting section includes section number 10000, section number 10001, section number 10002, and section number 10003. Section number 1 and section number 4 represent the input of mainline traffic volume and the input of on-ramp traffic volume, respectively. The data in the file obtained after the simulation run includes the trajectory records of all vehicles along the four paths, namely, vehicles upstream of the main line passing through the weaving zone directly enter the downstream of the weaving zone, vehicles upstream of the main line passing through the weaving zone enter the exit ramp, vehicles on the entrance ramp passing through the weaving zone enter the downstream of the main line, and vehicles on the entrance ramp passing through the weaving zone enter the exit ramp. 3.3. Training Results In VISSIM, the DQN agent underwent 500 learning and training sessions. From the training results, the DQN agent lacks learning experience, resulting in a decreasing trend in the reward obtained after each training. After a period of learning and training, the DQN agent accumulates a certain amount of learning experience, and the reward obtained shows a fluctuating upward trend. During the upward process, the early reward fluctuates greatly. When the DQN agent obtains enough learning experience, The results of each training in the later stage tend to be stable, and at this point, it can be determined that the convergence state has been reached, and finally the training model is obtained. It can be seen from the study of the speed distribution of each regional road section that, without control, since the input is the flow data in peak hours, the overall speed distribution in the weaving area, on ramp and speed limit area is in a lower range, which is consistent with the actual survey of traffic operation. In the case of fixed speed limit control, compared with the speed distribution map without control, it can be seen that the speed distribution of vehicles in weaving area and speed limit area has improved, but compared with the speed distribution map under variable speed limit guidance control, the speed distribution of vehicles in each area under variable speed limit guidance control has improved significantly, The variable speed limit guidance control based on the DQN algorithm can ensure that the main traffic flow operates at relatively stable speeds, thereby improving the traffic efficiency of urban expressways [7]. Figure 3. Vehicle Operation Trajectory under Uncontrolled Conditions Figure 4. Vehicle operation trajectory under fixed speed limit conditions Figure 5. Vehicle operation trajectory under DQN VSL Control Through the trajectory map under uncontrolled conditions, it can be seen that the trajectory of vehicle operation fluctuates significantly, and the phenomenon of "vacancy" is also more obvious [8]. That is, under uncontrolled conditions, due to the influence of weaving areas, the state of vehicle operation is extremely unstable, and the queuing phenomenon is also more serious. By observing the trajectory map under variable speed limit guidance control conditions, it can be 218 observed that after implementing the speed limit, the fluctuation amplitude of the vehicle's trajectory decreases, and the phenomenon of "vacancy" also decreases [9]. The traffic flow basically maintains a relatively uniform state of driving. 4. Analysis of Road Network Performance Indicators In VISSIM simulation, there are many evaluation indicators for road network performance. This article selects Total Travel Time, Average Delay, Average Stop, Average Travel Speed, Average Travel Time, and Number of Vehicles Passing through the weaving section as evaluation indicators, with the first four being the overall average evaluation indicators for the road network, The latter two serve as evaluation indicators for time-varying changes in the simulation process [10]. The simulation results of the three schemes are shown in Table 3. Table 3. Simulation evaluation indicators for overall performance of road network Method Total Travel Time Ave.Delay Ave.Travel Speed Uncontrolled 956.67 102.25 32.75 Fixed-speed 786.98 92.32 38.45 DQN 653.34 81.63 43.52 From the table, it can be seen that the overall performance of the road network is based on the order of uncontrolled simulation scheme, fixed value speed limit simulation scheme, and variable speed limit guidance control simulation scheme based on DQN algorithm [11]. The total travel time, average vehicle delay, and average vehicle stop times of the road network decrease in sequence, and the average driving speed of the weaving area gradually increases [12]. Compared to the uncontrolled condition, the total travel time of the road network is reduced by 18% with fixed speed limit control, and the average vehicle delay is reduced by 14.57%, The average number of stops per vehicle has decreased by 15%, and the average driving speed in the weaving area has increased by 10.66%; The variable speed limit guidance control based on the DQN algorithm reduces the total travel time of the road network by 35%, average vehicle delay by 29.64%, average vehicle stopping times by 31.63%, and average driving speed in weaving areas by 32.64% compared to without control. From this, it can be seen that the variable speed limit guidance control scheme based on the DQN algorithm has the best effect, verifying that the variable speed limit guidance control method proposed in this paper based on the DQN algorithm has good control effect [13]. 5. Conclusion Based on reinforcement learning algorithm, a variable speed limit guidance control method based on DQN algorithm was applied, and the key elements of the algorithm and experimental simulation were designed. On the basis of the VISSIM-COM interface of the simulation software, a secondary development program was implemented and the simulation experiment of variable speed limit guidance control was completed. Finally, the experimental results were combined with the road network performance indicators and vehicle operation trajectory [14], This indicates that the variable speed limit guidance control method based on the DQN algorithm can effectively improve the traffic efficiency of urban expressway weaving areas. References [1] Han Y, Hegyi A, Yuan Y, et al. Resolving freeway jam waves by discrete first-order model-based predictive control of variable speed limits[J]. Transportation Research Part C: Emerging Technologies, 2017, 77: 405-420. [2] Kattan L, Khondaker B, Derushkina O, et al. A probe-based variable speed limit system[J]. Journal of Intelligent Transportation Systems, 2015, 19(4): 339-354. [3] Zhibin, Li, Pan, et al. Optimal Mainline Variable Speed Limit Control to Improve Safety on Large-Scale Freeway Segments[J]. Computer Aided Civil & Infrastructure Engineering, 2016, 31(5):366-380. [4] Edara P, Sun.Evaluation of variable advisory speed limits in congested work zones[J].Journal of Transportation Safety & Security, 2017, 9(2): 123-145. [5] Highway Capacity Mannual.2010, Washington, D.C: Transportation Research Board, National Research Council. [6] Li Z, Liu P, Xu C, et al. Reinforcement learning-based variable speed limit control strategy to reduce traffic congestion at freeway recurrent bottlenecks[J]. IEEE transactions on intelligent transportation systems, 2017, 18(11): 3204-3217. [7] Wang C, Zhang J, Xu L, et al. A New Solution for Freeway Congestion: Cooperative Speed Limit Control Using Distributed Reinforcement Learning[J]. IEEE Access, 2019, 7: 41947-41957. [8] Zhu F, Ukkusuri S V. Accounting for dynamic speed limit control in a stochastic traffic environment: A reinforcement learning approach[J]. Transportation research part C: emerging technologies, 2014, 41: 30-47. [9] Lu C, Huang J, Gong J. Reinforcement Learning for Ramp Control: An Analysis of Learning Parameters[J]. Promet- Traffic&Transportation, 2016, 28(4): 371-381. [10] J. U. Duncombe, “Infrared navigation—Part I: An assessment of feasibility (Periodical style),” IEEE Trans. Electron Devices, vol. ED-11, pp. 34–39, Jan. 1959. [11] Wu Y, Tan H, Qin L, et al. Differential variable speed limits control for freeway recurrent bottlenecks via deep actor-critic algorithm[J]. Transportation research part C: emerging technologies, 2020, 117: 102649. [12] Ke Z, Li Z, Cao Z, et al. Enhancing transferability of deep reinforcement learning-based variable speed limit control using transfer learning[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, PP(99):1-12. [13] Muller E R , Carlson R C , Kraus W , et al. Microscopic simulation analysis of Mainstream Traffic Flow Control with Variable Speed Limits[C]// International IEEE Conference on Intelligent Transportation Systems. IEEE, 2013. [14] G. RG.R. Iordanidou, I. Papamichail, C. Roncoli, and M. Papageorgiou,“Feedback-based integrated motorway traffic flow control with delay balancing,” IEEE Trans. Intell. Transp. Syst., vol. 18, no. 9, pp. 2319-2329,Sep. 2017.