Modelling And Analysis Of The Evolutionary Dynamics Of Population Behavior For Stochastic Games

Cooperative behavior exists widely in all levels of living systems and plays a vital role in promoting the evolution of species and the development of human society.Understanding how cooperative behavior emerges among selfish individuals has been an important and challenging problem.A large number of past research results have provided effective solutions to the common cooperation problems in the real world,but these studies often assume that the payoff matrix is fixed in the interaction process of agents.This assumption is an oversimplification of the real interaction scenario of individuals and ignores the time-variability of the interaction environment of agents,especially the environmental changes caused by agent behaviors.The well-known tragedy of the Commons in economics is a typical example.The excessive grazing behavior of herdsmen will lead to the degradation of the public grassland,making herdsmen face the problem of resource shortage in the following days,and moderate grazing is conducive to the sustainable utilization of the grassland resources for herdsmen.Under such scenarios,what kind of evolutionary dynamics of population behavior will present has become a hot topic.The stochastic game can describe the interplay between agent behaviors and the environmental state,which has become an important framework for scholars to study the evolution of population behavior in a dynamic environment.In this paper,the pair-approximation method in statistical physics is novelly used to model the evolutionary dynamics of population behavior in stochastic games,and the Monte Carlo method is used to carry out agent-based simulations to further analyze the evolution and equilibrium of the stochastic game.The research content of this paper is divided into the following two parts:The first part includes proposing a multi-agent stochastic game model and modelling the evolutionary dynamics of population behavior.This paper proposes a multi-agent stochastic game model based on the relationship between agent behaviors and the environment in the real world.In this model,the structure of the population corresponds to a complete graph,each agent occupies a node on the graph,every two agents are connected by an edge,and each edge is associated with a 2-agent stochastic game with symmetric state transition rules.In the interaction of each time step,each agent chooses an action according to its own strategy to play games with all its neighbors.The agent updates its strategy through the Q-learning algorithm.The state transition of the stochastic game associated with each edge is driven by the joint action of the connected agents and the corresponding current state.Based on this multi-agent stochastic game,this paper models the evolutionary dynamics of population behavior.In this paper,the inapplicability of the mean-field theory in stochastic games is analyzed,the mean-field theory is often used in modelling learning dynamics,and the pair-approximation method is used to track the evolution of different data distributions of different agents.In this paper,the dynamical equations describing the evolution of the environmental state of the population and the evolution of the conditional probability distribution of Q-value vector pairs in each state are derived respectively,and a partial differential equation is obtained to describe the evolution of the probability distribution of pairs.By this dynamical model,the evolution of the population behavior and the evolution of the environmental state in stochastic games can be accurately predicted.The second part includes the experimental validation of the dynamical model and the analysis of the evolution and equilibrium of the population stochastic game.In order to validate the applicability of the dynamical model and reveal the characteristics of the evolution of population behavior and internal mechanism in the stochastic game,a lot of agent-based simulations are conducted using the Monte Carlo method in this paper.This paper verifies the applicability of the dynamical model under different games,initial conditions,state transition rules,population size and algorithm parameters,and reveals the effects of these different factors on the evolution of population behavior through experiments.In this paper,we find that under certain conditions,even though neither of the two games alone supports the emergence of cooperation,the transition between the two games can significantly promote the evolution of cooperation,which means that myopic reinforcement learning agents can also learn to cooperate in an ever-changing environment.The Q-learning dynamics model based on the pair-approximation method proposed in this paper is an application of the statistical physics method in modelling the reinforcement learning dynamics and reveals the relationship between statistical physics and multi-agent reinforcement learning.The key to the pair-approximation method in this paper is how to solve the probability distribution of pairs and the temporal evolution of this distribution,this method can be further applied to model the evolutionary dynamics of population behavior in more complex interaction scenarios,and it will also bring inspiration to the relevant work aiming at modelling the learning dynamics.Through a lot of experimental results,this study finds that the state transition mechanism plays an important role in promoting the evolution of cooperation,thus providing theoretical guidance for solving problems such as global warming and public resource management.