Research On Fps Games Based On Deep Reinforcement Learning

Deep reinforcement learning which combines the advantages of deep learning and reinforcement learning has recently made great breakthroughs in several fields including games,robotics,autonomous driving and recommendation systems.Games have the advantage of simple and fast data sampling,which greatly facilitates the research of deep reinforcement learning.Although deep reinforcement learning has shown superior performance than human beings in Atari 2600 games and board games,it still faces important challenges in complex tasks such as First-person Shooting(FPS)games.Based on the ViZDoom platform,this paper analyzes the characteristics of FPS games,proposes the agent decomposition scheme,and adopts the multi-task learning method.This paper mainly focuses on the research as follows:(1)FPS games usually have complex discrete action space,diverse tasks and difficulty to distribute game rewards.This paper proposes the agent decomposition scheme to solve these problems.The agent decomposition disintegrates an agent from three aspects of action space,task type and reward allocation,so as to solve more complex tasks.Firstly,this paper proposes a method of semantic action space decomposition,which decomposes the original action space into several action subspaces according to the combination rules and practical significance of actions.Each action subspace corresponds to a Q network branch,reducing the complexity of the original action space.Secondly,this paper proposes a task decomposition method,which divides the compound task into two sub-tasks,navigation and attack.Each sub-task includes several related Q network branches,which reduces the learning difficulty of the compound task.Thirdly,this paper proposes a reward decomposition method,which allocates all rewards to different Q network branches according to the distribution matrix.Each Q network branch only relies on the rewards obtained by itself to update,so as to promote the agent to learn more pertinently.(2)In FPS games such as ViZDoom,the partially observable 3D environment simulates the real world preferably,making it more difficult to extract features from the original pixels.In this paper,the multi-task learning method is used to solve this problem.Based on the Q network,an auxiliary classification network sharing the convolution layers is added.This network is used to identify whether there are enemies and resources in the image.The joint training of Q network and classification network enhances the ability of agent to understand the environment.In addition,the output of the classification network can also be used in sub-task scheduling.Experiments show that the proposed methods including action space decomposition,task decomposition and reward decomposition can improve the agent's performance in ViZDoom.At the same time,the overall scheme of this paper is better than other schemes.