Intelligent Decision-making Method For Unit Micromangement In Real-time Strategy Game

The unit micromanagement of Real-time strategy(RTS)game is a great challenge in artificial intelligence research research,which aims to gracefully control multiple in-volved units to win the combat.Existing works mainly include search methods which rely on online search,and multi-agent deep reinforcement learning(DRL)which rely on environmental interactive learning.However,in the large-scale combat scenarios,large action space of multiple agents makes it very difficult for search methods to find the optimal actions in time,and it is difficult to train a general DRL model.The success of AlphaGo and AlphaGoZero has demonstrated the effectiveness of combining deep learning and Monte Carlo Tree Search(MCTS).And well-trained learning model can guide the search process towards the optimal solution.Inspired by this,we first formu-late the action prediction into an efficient end-2-end learning problem,then incorporate the well-trained learning model into some typical search methods.In order to complete the end-2-end learning of joint action,we proposed a suit of novel feature maps to represent the game state,and several action maps to represent the joint action of own units.And we proposed joint policy network(JPN),an encoder-decoder convolutional neural network,to complete such action prediction for own units simultaneously.Specially,unlike multi-agent DRL methods which regard units as indi-viduals,and perform a forward pass for each unit.The proposed JPN regards own units as a group,and performs one forward pass for all of own units together.Further,we incorporated the proposed JPN into some typical search methods,including PGS,POE and SSS+.Our modified search methods(PGS w/JPN,POE w/JPN and SSS+w/JPN)adopt similar ideas,i.e.,utilizing the prior probabilities produced by JPN to generate a good initial solution,then guide the search process toward the optimal solution.Spe-cially,different from the family of MCTS,our modified search methods do not have the time overhead of building up a search tree and frequently using the learning model.In order to verify the effectiveness of the proposed method,this dissertation con-structed several datasets which covering the baseline combat scenarios,and conducted extensive experimental studies on SparCraft,StarCraft:BroodWar and gym-starcraft.First,the results demonstrate the effectiveness of the proposed network architecture and loss function.The proposed JPN can defeat PGS,and its time-consuming is not affected by the scale of the combat scenarios.When the number of controlled units reaches 8 and above,its time consumption is significantly lower than multi-agent DRL methods.Second,our modified search methods can defeat GAB and the built-in AI,and SSS+w/JPN can substantially outperform state-of-the-art SAB.Finally,the proposed JPN and modified search methods are more suitable for large-scale battle scenarios,and the proposed JPN and combination mechanism can significantly enhance PGS,POE and SSS+.In summary,this dissertation proposed an intelligent decision-making method by combining deep learning and search methods.This method has achieved good results on various indicators and has good practical application value.