Credit Assignment Techniques In Cooperative Multiagent System

The credit assignment technique involves determining the contribution of components to the overall system performance.The success of the reinforcement learning is inseparable from the credit assignment.In the multiagent reinforcement learning,the credit assignment is mainly the distribution of global reward signals for agents in a cooperative environment.Studying the credit assignment mechanism aims to solve the problem of inaccurate update of agents under the cooperative multiagent deep reinforcement learning,and to improve the learning efficiency and cooperation degree of multiagent systems.Therefore,multiagent credit assignment is the key to solving the cooperative multiagent coordination problem.However,most of the previous researches use the difference reward to reduce the noise of actions,which is computationally expensive to specify the default state or action and set in simple environments.In this paper,two real-world scenarios of the electricity broker design in retail market and large-scale home energy management under smart grid are investigated.We apply and improve the existing difference reward technology in the two scenarios to greatly improves the coordination of multiagent systems and solve the practical problems in the smart grid.On the other hand,the latest multiagent credit assignment algorithms based on deep learning also have design or theoretical deficiencies.COMA takes advantage of the weighted average of all action values to avoid the above problem,but it uses on-policy to result in lower sampling efficiency and its critic structure is not suitable for largescale multiagent systems.VDN and QMIX based on value decomposition constraint the relationship and generalization of individual Q-values and overall Q-values.At the same time,these techniques did not consider heterogeneous multiagent systems.Combining with the latest advances in deep learning,this paper proposes a new method of credit assignment,which effectively solves the coordination problem in heterogeneous multi-agent systems.