Design Of Reinforcement Learnig Controller And Assisted Driving System For Heavy-haul Train

With the advancement of science and technology,the development of China’s transportation industry has been particularly prominent.The rail transportation industry has made leapfrog progress in all aspects,and has accumulated a large number of leading-edge technologies in the world.In the background of rapid national economic development,the rail transportation is busier than ever before.As an important route for the transportation of bulk commodities,heavy-haul railways have long distances and changeable environments.Faced with such an environment,drivers are extremely prone to fatigue.In this paper,the HXD1 locomotive hauling 10,000 tons of freight cars on the Daqin line is researched,and a set of auxiliary driving system for heavy-haul trains is trained through the reinforcement learning method to relieve driver fatigue,ensure train safety,and improve train transportation efficiency.Aiming at the operation efficiency of heavy-haul trains,this paper uses two reinforcement learning algorithms,TD3 and PPO,to learn control strategies.First,analyze and model the mechanism model of the heavy-haul train as the basis of subsequent experiments;then divide the operation process of the heavy-haul train into three parts: traction startup,cruise control,and braking phrase in case the agent learned little control strategies in long-term reinforcement learning;the state space is selected based on the conditions that the actual train control needs to refer to,the continuous action space is determined according to the traction characteristics of the HXD1 locomotive;the reward function is designed based on the goals of safety,stability and efficiency.According to the above steps,two kinds of reinforcement learning agents are trained.The simulation results show that the state and reward function designed in this paper can enable the agent to learn a control strategy for driving a heavy-haul train efficiently.Aiming at the stability problem of heavy-haul train operation,this paper designs a reinforcement learning training program based on expert supervision to make the reinforcemnet learning agent drive the train more stably.First,the behavior clone of expert driving data is carried out through the recurrent neural network.The cloned strategy network is used as the expert network to supervise the training of reinforcement learning,which achieves the effect of accelerating the training;and by adding the constraint of the control force change range per step,the control force change is smoother;and train a cruise control strategy and braking strategy that are not sensitive to the environmental switching speed by randomly initializing the initial speed of the cruise phase and the braking phase;finally,design soft switching constraints at the segment points of startup phrase,cruise control phrase,and braking phrase,to reduce the jump of control force caused by the switching of control conditions.The simulation results show that the controller based on the reinforcement learning training program supervised by experts has more stable control force and can ensure the safe operation of trains.Since there is not heavy-haul train simulator for observation and debugging in the reinforcement learning environment yet,and the actual operation data of heavy-haul trains are lacking either.this paper uses Qt to design and build a heavy-haul train simulation platform,observing the effect of train operation with simulator,and it also can assist in debugging the possible problems of the algorithm,and used to generate expert driving operating data to assist expert strategy network training.