| Collective behavior modeling is a method to better understand the mechanisms that govern collective animal behaviors.Traditional modeling methods based on rules and motion models can describe some actual collective behavior,but the preset rules and motion models are abstracted from human prior knowledge,thus limiting the expressiveness of these models and their ability to capture collective behavior.Deep neural networks have the ability to express complex functions,and in recent years,Deep Reinforcement Learning(DRL)has become an important research direction in collective behavior modeling.However,current DRL-based modeling methods generally encode rules in some form into the reward function,and are still limited by rules.To construct a DRL-based collective behavior modeling method that does not rely on prior knowledge,this paper conducts related research.The research contents are as follows:(1)In response to the problem that both traditional modeling methods and learning-based modeling methods are limited by rules,this paper proposes a fish school collective behavior modeling method based on Deep Q-Networks(DQN).This method first uses the angle between the fish individual’s movement direction and the average movement direction of surrounding neighbors to represent the individual’s state(continuous value)and discretizes the turning angle to represent its action.Then,a reward function based on the change of the number of neighbors is designed,and experiments are conducted in both single-learner and multi-learner modes.The experimental results show that the proposed method can successfully model the individual decision-making model that forms effective collective behavior,with good application prospects and research significance.Particularly noteworthy,this study first discovered a "binary strategy" that can make individuals form collective behavior,which has important implications for solving practical problems such as swarm control.(2)This paper proposes a fish school collective behavior modeling method based on Generative Adversarial Imitation Learning(GAIL)to address the problem of reward function design in DRL-based fish school collective behavior modeling.The goal is to investigate whether individual movement policies that form collective behavior can be directly modeled from expert collective behavior trajectories without designing reward functions.This method first uses the Vicsek model to obtain expert collective behavior trajectories for training,then designs the raw observation representation of individuals to make their perception closer to reality,and finally conducts experiments in single-learner and multi-learner modes.Experimental results show that the proposed method can directly model individual decision-making models that form effective collective behavior,overcome the problem of reward function design in DRL-based modeling methods,and improve the efficiency and accuracy of collective behavior modeling.(3)Currently,most learning-based methods for modeling collective behavior place the cluster in an environment with periodic boundary conditions and do not consider collisions between individuals.To model the behavior of real farmed fish school collective behavior,this paper proposes a closed environment fish collective behavior modeling method based on GAIL.Firstly,a closed environment collective behavior simulation model is used to obtain expert trajectories that approximate the behavior of real farmed fish clusters.Then,a feature extraction function is designed and used to address the difficulties in learning caused by the complexity and high dimensionality of individual raw observations.Finally,relevant experiments are carried out.The experimental results show that the proposed method is successfully applied to modeling collective behavior in a complex closed environment,and the individual decision models obtained can enable learners to form two different modes of collective behavior in the closed environment,fully demonstrating the effectiveness and feasibility of the proposed method. |
