Study On Micro-scale Pedestrian Simulation Using Reinforcement Learning

The public safety problem has aroused widespread attention,while the pedestrian simulation is a useful tool that can not only realize the discovery or prediction of the potential public safety problem but also can conduct pre-evaluation of the plans established to deal with this problem.Compared with evacuation drills,pedestrian simulation has the advantage both in cost and flexibility,besides,it could be quickly constructed facing different scales scenarios.More importantly,it can be accurately reproduced the different movement behavior due to the change of the mental state in various situations through the improvement of the simulation algorithm.Based on that,pedestrian simulation has become a research hotspot in recent years.However,traditional pedestrian simulation methods usually face the following problems: 1)virtual pedestrians lack foresight during the moving process,so that chaotic and oscillatory motions are generated in complex scenarios;2)most of the current methods require sophisticated mathematical formulas to describe pedestrian's behavior,which is difficult to design and usually has certain limitations;3)virtual pedestrians lack selfadaptability during movement,which leads to that pedestrians are impossible to deal with the encountered trouble;4)most of the algorithms apply passive information,such as neighbor's position and velocity when planning the optimal pedestrian velocity,which will further lead to the inefficiency of the simulation process.In recent years,machine learning,especially deep learning,has achieved remarkable performance in many fields.As one of the branches of machine learning,reinforcement learning is a learning method based on the Markov decision process to optimize the longterm rewards.By cleverly designing the reward functions,reinforcement learning can guide the agent to take the best action in a certain state.Considering the Markov properties in pedestrian simulation,it is possible to design corresponding reward functions for virtual pedestrians,combined with reinforcement learning,to learning the best behavior through deep learning or adaptive learning pattern.However,the integration of reinforcement learning and pedestrian simulation is still in its infancy and deserves further exploration.To increase the foresight of pedestrians in the movement process and adaptability to complex scenarios in the micro-scale simulation,we explored a simulation method that combines the traditional pedestrian simulation and reinforcement learning methods.The pedestrian simulation designed in this dissertation is a part of evacuation simulation,where each pedestrian has a clear destination and desires to arrive as soon as possible,so it is different from simulations such as in shopping malls or queuing in the canteen.To prove the applicability,we summarize the research in different scenarios and application purposes,also conducts a relatively comprehensive exploration of the combination of existing reinforcement learning methods and pedestrian simulation.Based on analyzing and discussing the existing research,we show some performances and experiences in this field.Specifically,assuming that with the support of spatial positioning and indoor modeling,we proposed our research content which applies GIS simulation platform and reinforcement learning framework to solve the predicament in traditional pedestrian simulation,and discuss its effectiveness and applicability.According to the different research spaces,we further divide the research into local space and global space.In the local space,our goal is to guide virtual pedestrians to make decisions in advance for future collision through deep reinforcement learning where a convolutional neural network is used to abstract the mapping between pedestrian's state and action.To verify the impact of reinforcement learning on pedestrian movement behavior,our content splits into two parts,namely policy-based reinforcement learning for local motion trajectory smoothing and value-based reinforcement learning for multi-exit evacuation simulation,in which trajectory smoothing aims to study the optimal form of pedestrian movement in an ideal and orderly environment,and multi-exit evacuation simulation aims to study evacuation routes for pedestrians facing different exit attributes and crowded environments.In the global space,we propose an evacuation simulation oriented to complex scenarios,in which pedestrians have adaptive learning capabilities to cope with the dynamic and risky environment.Besides,a communication mechanism is added as a part of the reward function to improve stability during the adaptive learning process.Finally,the experiment proves that our method is not only more flexible in design,but also efficient and scalable in performances.Overall,our contributions are as follows: 1)proposing a new pedestrian simulation method that integrated traditional local collision method and reinforcement learning.The advantage is that traditional methods can perfectly avoid collision for each pedestrian in each frame,while reinforcement learning method makes the pedestrian more intelligent due to injecting adaptive learning capabilities;2)simulating the pedestrian's behavior through the reward functions,which will be a simpler and feasible design compared with mathematical formulas.Moreover,our method has stronger expansion and generalization capabilities;3)setting up communication reward function via adding positive communication mechanism between neighbors,which is helpful for pedestrians to have a more comprehensive understanding of the surrounding environment during the movement.