Research And Implementation Of Micro Real-time Strategy Game Based On Deep Reinforcement Learning

In future wars,the traditional combat using single equipment will be replaced by multi equipment.Collaborative electronic combat uses computer and communication technology to connect existing electronic combat systems into a network in order to maximize the use of existing technology and tactical resources,improve the system’s perception,attack and protection abilities so that ultimately achieve the goal of cooperation and improve the efficiency of existing electronic combat systems.The centralized learning technology is difficult to adapt to the heterogeneous,complex,dynamic and large-scale real-time systems.However,the multi-agent reinforcement learning technology has the characteristics of autonomy,distribution,coordination,self-organization and self-learning,which is suitable to build a cooperative and competitive multi-agent system with strong robustness and reliability.This thesis is dedicated to using deep reinforcement learning technology to solve problems in multi-agent antagonism systems.This study mainly focuses on the mini game scenarios and the full game scenarios in real-time strategy games:1)In mini game scenarios,the traditional action-value function has been widely used,but action-value function is difficult to adapt to the real-time strategy game systems,because the number of agents can change at any time.In order to make the multi-agent architecture more scalable,we explore a multi-agent reinforcement learning framework for real-time strategy game that avoids the use of action-value functions,i.e.using the state-value function shared globally by all agents and multiple switchable actors in the Actor-Critic framework.We also design a suitable global reward function for this framework.We then propose the algorithm without human knowledge which can work for semi-markov decision process where rewards cannot be received until actions last for a while,which is more suitable for real-world scenarios.2)In full game scenarios,the problem has a higher state and action space than mini game scenarios.Based on the presented reinforcement learning framework in mini game scenarios,this thesis proposes the counterfactual advantage function and the league learning method respectively to solve the two problems,that is,lack of cooperation between agents and training is more likely to fall into local optimum in full game scenarios.3)This thesis gives the details of the design and the implementation of the reinforcement learning system.To evaluate the performance of our method,this thesis conducts mini game and full game experiments on a simplified real-time strategy game,!RTS.The results show that the AI trained by this method is highly competitive against strong baseline AIs.