Intelligent Pedestrian Simulation Research Based On Deep Reinforcement Learning

Pedestrian simulation has an important position in public safety research,but how to make the behavior of simulated individuals close to reality is a difficult problem.The application of deep reinforcement learning in the field of pedestrian simulation has high flexibility,strong scalability,simple model,and can improve the reality of simulation,which is a new development direction in the field of pedestrian simulation.At present,deep reinforcement learning has been widely used in agent simulation,but due to the lack of related software tools,agent simulation still stays at the mass point level and can-not be applied to the field of pedestrian simulation.In recent years,with the emergence of related tools,it has become possible to apply deep reinforcement learning to the field of pedestrian simulation.Therefore,we propose a decision model based on deep reinforcement learning for a single pedestrian,and establish a general pedestrian simulation framework based on deep reinforcement learning.Furthermore,deep reinforcement learning is applied to multiple pedestrians,and a multi-agent distributed distributional deep deterministic policy gradient algorithm is proposed for multi-pedestrian scenarios to simulate real pedestrian behavior in simulation scenarios with multiple roles.By introducing deep reinforcement learning into the field of pedestrian simulation,we establish a single pedestrian decision model based on deep reinforcement learning,and further propose a pedestrian simulation framework based on deep reinforcement learning.Most existing single-agent deep reinforcement learning algorithms only consider environmental observation information at a single moment,which makes pedestrians lack memory and slow convergence.Therefore,a fixed-size sliding time window is introduced to enable pedestrians to have a certain memory ability.Experiments show that pedestrian based on deep reinforcement learning decision-making model can produce real post-transcendence behavior in emergencies,and can perceive the situation,making the motion trajectory more realistic and reasonable.The introduction of a fixed-size sliding time window can speed up the convergence of the decision-making model.In complex multi-agent scenarios,the existing multi-agent deep reinforcement learning algorithms have slow convergence speed,and the stability of the algorithms cannot be guaranteed.We propose a new multi-agent deep reinforcement learning algorithm,which we call it Multi-Agent Distributed Distributional Deep Deterministic Policy Gradient(MA-D4PG).We adapt the idea of value distribution to the multi-agent scenario,retain the complete distribution information of expected return,so that agents can obtain a more stable and effective learning signal;We also introduce multi-step return to improve the stability of the algorithm;In addition,we use a distributed data generation framework to decouple empirical data generation and network update,so that more computing resources is utilized to make the convergence progress faster.Experiment show that proposed method has better stability and convergence speed in multiple continuous/discrete controlled multi-agent scenarios and the decision-making ability of agents has also been significantly enhanced.In the multi-pedestrian simulation scenario,security personnels based on the MA-D4PG algorithm decision-making model can individually adjust the speed and direction of movement to cooperate to capture dynamic targets,which reflects the authenticity and rationality of the simulation effect.