A Research Of Hierarchical Multi-agents Deep Reinforcement Learning For Action Game

Game AI is the combination of artificial intelligence and the game field,and is an important experimental carrier for reinforcement learning(RL).RL focuses on the problem of sequential decision making,which coincides with action selection in games.In this field,many remarkable research results have appeared in recent years,such as DQN which play Atari game near human level,AlphaGO which defeat the world top Go players,OpenAI Five which win a top Dota2 e-sport team.Traditional RL uses low-dimensional data.Also,its state and action space is relatively small.But now it faces more and more challenges when facing high-dimensional state and action space,sparse and delayed rewards,non-stable environment in multi-agent system.Based on a multiplayer football game,this paper proposes a hierarchical multi-agent RL algorithm with inter communication mechanism(HMARL-ICM)to solve the cooperation and competition between football players.Non-stable environment is an unavoidable problem in multi-agent RL.The reason is that other agents will also affect the environment,which may cause the deviation of understanding of environment state.Following the agent-independent method,this paper designs a decentralized training and an inter communication mechanism to make environment stable.And this idea also promotes the cooperative behavior between multiple players.Multiplayer football game is complicated and its episode is a bit long,which makes the rewards are spare and delayed.To solve this problem,this article constructs a hierarchical Actor-Critic network based on temporal abstraction theory,which network is similar to the structure of decision-making by brain and action by trunk.The core algorithm of high and low level is Fast-PPO,which releases the restrictions of proximal policy optimization to get improved.Applying the algorithm HMARL-ICM on a 3v3 multiplayer football game which is made by unity,this paper gets better results and faster convergence than other advanced RL algorithms.Agents can cope with long decision making in complex games based on hierarchical structure.And the inter communication mechanism enables multiple agents to share their information,which leads to a variety of excellent football strategies.