Research On Deep Reinforcement Learning Transfer Methods In Multi-Agent System

Reinforcement learning technology is used to describe and solve the problem that agents learn strategies in the process of interacting with the environment in order to maximize returns or achieve specific goals.Deep reinforcement learning is a further development of reinforcement learning to cope with more complex environments.Multi-agent environment is a common complex environment,in which the training cost of deep reinforcement learning is usually very high,which greatly hinders the technical development and application promotion of deep reinforcement learning.Transfer technology is a technology that can effectively reduce the training cost of multi-agent deep reinforcement learning.It refers to applying the knowledge or pattern learned in one domain or task to other domains or problems.In addition,transfer technology is also one of the key technologies for the smooth application of multi-agent deep reinforcement learning model from the experimental environment to the actual environment.Therefore,it is very necessary to research the transfer of multi-agent deep reinforcement learning.The transfer of multi-agent deep reinforcement learning can be divided into three levels:team level,individual level and local sub-strategy level.Transferring team strategy is often used to realize the change of team scale,but the existing transfer method is difficult to cope with the dynamic change of agent scale environment.Transferring individual strategy is often used to accelerate the training and improving the performance of the algorithm,but the traditional transfer method has limited improvement in the training speed and is difficult to ensure the effectiveness of the transferred knowledge.Transference of local sub-strategies can reduce the training cost,but it is difficult for the algorithm to decompose sub-strategies from the total strategy for transferring under the traditional neural network structure.This paper makes an in-depth study of the above three issues,and the main work and innovation points are as follows:(1)Aiming at the problem that existing transfer methods cannot cope with the dynamic change of agent scale,this paper proposes Sequence to Sequence Multi-Agent Reinforcement Learning(SMARL)algorithm.By transferring knowledge among internal agents and reducing the correlation between agent scale and algorithm,the algorithm improves its adaptability to the change of agent scale.Experimental results show that the proposed algorithm is superior to the baseline algorithm in both transferability and training efficiency,and its performance after transferring is at least 3 times that of the baseline algorithm when the number of agents changes.(2)To solve the problem that traditional transfer method has limited improvement in training speed and is difficult to guarantee the effectiveness of the transferred knowledge,this paper proposes Supervised Reinforcement Learning(SRL)algorithm.Through the way of "supervised pre-training ? knowledge transferring? reinforcement learning",to accelerate the purpose of training.Experiments show that in the case of single agent,the training time of this algorithm is at least 25% shorter than that of other methods.In the case of multiple agents,this algorithm can set a good "start" for agents to obtain more rewards.(3)In view of the problem that it is difficult for the algorithm to decompose sub-strategies from the total strategy for transferring under the traditional neural network structure,this paper introduces the idea of "high cohesion and low coupling" into deep reinforcement learning and proposes a modular deep reinforcement learning model.In this model,the transferring and reuse of local modules and specific strategies between agents is realized by dividing the whole network and decomposing the overall strategy.Experiments show that this model is much better than the traditional deep reinforcement learning algorithm in terms of performance,transferability and generalization.