A Multi-agent Reinforcement Learning Algorithm Based On Stackelberg Game

Multi-agent systems(MASs)arise from diverse fields.Multi-agent reinforcement learning(MARL)has been paid much attention in recent research,due to its wide applications in various engineering systems.However,most of previous algorithms in both of MASs and MARL assumed that all the agents make decisions simultaneously.This does not fit into many practical situations.Thus,we transferred Multi-agent system involving a major agent(leader)and a number of minor agents(followers)into Stackelberg games.The agents are divided into leaders and followers in term of roles,and all the agents give the strategies in order.In this paper,firstly,a novel multi-agent Q-learning algorithm based on Stackelberg game is proposed,called Stackelberg Q-learning.Moreover,in recent years,knowledge transfer has been applied into multi-agent systems(MASs)to improve the scalabilities of the systems.Stackelberg Q-learning with VFT is proposed as an improvement of Stackelberg Q-learning.In this paper,for case study,anti-jamming power control problems of secondary users in a large-scale cooperative cognitive radio network attacked by a smart jammer are formulated as multi-agent systems.Compared with Nash Q-learning,our algorithm is more suitable to deal with multi-agent systems with hierarchical structures and also has higher speed of convergence.Moreover,the scalability of MASs is improved by VTF.In addition,we also discussed some detailed problems,such as the effects of changing parameters.