Multi-agent Coordination Through Decoupled Reinforcement Social Learning

In this paper we focus on the multi-agent coordination problem.Multi-agent systems(MASs)refer to the computing system composed of multiple independent agents.Coordination is the key to improve the overall profit.The goal of multi-agent coordination is to generate a consistency strategy.And the dominant strategy represents the social convention.For large-scale distributed multi-agent systems,since there is no center node in the system,it is difficult to directly generate coordination strategy before interaction.Therefore,how to design efficient strategies to promote convention emergence in MASs with large convention space is of great significance.For the above problems,we model multi-agent coordination control problem as coordination game,propose a decoupled social reinforcement learning framework,and design learning strategies based on state space decomposition.We apply our approaches to a language coordination game in which agents need to coordinate on a dominant lexicon for efficient communication.Each lexicon which maps each concept to a single word is an alternative convention.To update the lexicon directly is a single-state convention learning problem.The state is the system itself and remains unchanged during the learning process.The action is the whole lexicon.The single-state learning strategy cannot guarantee the convention convergence as the convention space increases.Thus by modeling each lexicon as a Markov strategy representation,the original single-state convention learning problem can be transformed into a multi-state multi-agent coordination problem.The state is the concept and the action is the word.Specifically we propose two learning strategies,multiple-Q and multiple-R,and also propose incorporating teacher-student mechanism and network rewring on top of the learning strategies to accelerate lexicon convergence speed.Extensive experiments verify that our approaches outperform the state-of-the-art approaches in terms of convergence efficiency,convention quality and scalability.