Application Of Pruning Method In Multi-Agent Reinforcement Learning

In recent years,with the vigorous development of deep neural network,many breakthroughs have been made in the field of reinforcement learning,which makes Multi-Agent Reinforcement Learning(MARL)receive extensive attention.However,marl also faces many challenges,such as huge joint action space,non-stationary,partial observation of the environment.Among them,the huge space of joint action will lead to a huge amount of calculation.How to reduce the amount of calculation has always been a research hotspot.Model pruning can reduce the parameters of neural network model and reduce the amount of calculation.Moreover,some studies have shown that model pruning method can be applied not only in the field of traditional image processing,but also in natural language processing and reinforcement learning.Inspired by this,this paper studies the application of pruning method in Multi-Agent Reinforcement Learning.Firstly,the application of pruning method in Multi-Agent Reinforcement Learning is studied.The experimental framework selects Multi-Agent Reinforcement Learning Framework SMAC,and adopts global pruning method and layer-wise pruning method to process the neural network model.The experimental results show that the pruning method can be well applied to Multi-Agent Reinforcement Learning,which can significantly reduce the amount of network model parameters and maintain the performance close to the unpruned network.Then,aiming at the problem that the accuracy of the network model will drop sharply with the imbalance of the network structure when the global pruning method is in the later stage of the pruning process,and the insufficient utilization of the network feature information by the traditional layer-wise pruning method,we propose the Network-Structure-based Iterative Pruning Method(NS-IPM).The experimental results show that NS-IPM performs well,which can not only avoid the structural imbalance of global pruning method,but also make full use of network feature information.However,as a pruning method,the effectiveness of NS-IPM still needs to be verified in the field of image processing.Therefore,we conduct experiments on image recognition dataset cifar-10,using vgg-16 and resnet-20 as network models,and compare NS-IPM with global pruning method and traditional layer-wise pruning method respectively.In vgg-16 model,When the compression rate is nearly twice as high,the accuracy is still 3.6% higher than the layered pruning method with single pruning rate,and the overall performance is better than the global pruning method.When the compression rate reaches more than 98.85%,the accuracy of resnet-20 model is 20% higher than that of the layered method with single pruning rate,which is close to the global pruning method.This proves that NS-IPM has obvious advantages compared with traditional layer-wise pruning method,and provides an alternative solution for the limited application of traditional layer-wise pruning method.More importantly,the experimental results also prove that NS-IPM method not only performs well in Multi-Agent Reinforcement Learning scene,but also can be well applied to the field of image recognition.This shows that NS-IPM is a highly applicable and effective pruning method,which provides a new idea and method for the research of Multi-Agent Reinforcement Learning.