Learning Predictive State Representations Model For Multi-agent System

Sequential prediction is the core issue of agent technology.It mainly considers how to establish an accurate dynamic system model in a dynamic environLent full of uncertainties,various disturbances,complex and changeable,realize accurate prediction of multi-agent action-observation sequence and establish its optimal decision-making model.Among many sequential prediction models,the Predictive State Representations(PSR)represents the state of the system by action-observation sequence vectors,and then realizes the accurate prediction of the probability of future events.PSR has excellent characteristics such as strong representation ability,easy modeling and learning,and effectively overcomes the shortcomings of traditional models in dealing with sequential decision-making problems of agents.It can be widely used to solve single agent prediction and decision-making problems.At present,the algorithm of learning PSR model has been relatively perfect,but its reliability and efficiency need to be improved,and these studies usually consider how to model a single agent.Moreover,the research of PSR model has not yet involved the multi-agent modelling problem,and most of the research work is still focused on the improvement of model learning efficiency and scalability.It is very difficult to learn a multi-agent PSR model,especially with the increasing number of agents and the complexity of the problem domain,the difficulty of modeling is greatly increased.At the same time,with the progress of data storage and processing technology,most multi-agent systems accumulate a large amount of interactive data when they interact with the system.When interactive data is represented in high-dimensional space,it shows obvious sparse phenomenorr How to make full use ofthese data to build agent prediction model and improve the decision-making ability of multi-agent will be a problem with broad application prospects.Therefore,this paper mainly promotes the PSR model of multi-agent on the basis of the PSR model of single agent,and begins to solve the challenging multi-agent modeling problem,especially from a large number of multi-agent interactive data to learn the PSR model:Aiming at the problems of poor accuracy and low efficiency in the modeling of large-scale complex multi-agent systems,the traditional predictive state representation model is based on the optimization technique-based multi-agent prediction state representation model modeling method.This method promotes the PSR model of multi-agent based on the PSR model of single agent,and then transforms the core test set discovery and model parameter learning of multi-agent PSR model into global optimization problem,and solves it by optimization technology.Specifically,the problem of the discovery of the PSR model is formalized into a convex optimization problem.Considering the global sparsity of the system dynamic matrix,the convex optimization problem is transformed into the Lasso problem.Finally,the alternating direction multiplier method(ADMM)is used to solve the problem.When the system PSR model core test set is obtained,the PSR model parameters are ob-tained together,and thus the system PSR model can be obtained immediately.In addition,the ADMM optimization technique used in this paper has global convergence guarantee,and the global sparse optimization method of learning prediction state representation model proposed in this chapter does not need to preset the size of the core test set.The system state matrix for the traditional predictive state representation model indicates the insufficiency of the multi-agent system dynamics.This paper studies a highdimensional system dynamics matrix,ie tensor,for learning multi-agent PSR models,and proposes tensor-based decomposition.The multi-agent predictive state representation model.Based on the TPSR model of single agent,a framework for learning two agents and multi-agent PSR using tensor decomposition is proposed.Firstly,how to apply the traditional tensor decomposition method to PSR learming of two agents,and then It extends to the case of multiple agents.The use of tensor decomposition to learn the framework of two agents and multi-agent PSR solves the problem of multi-agent modeling,especially the PSR model obtained from a large number of multi-agent interaction data.The difficulty of modeling multi-agent system increases with the number of agents,which ultimately makes it difficult to obtain a feasible prediction state representation model.This paper proposes a tensor-based optimization technique to solve the multi-agent prediction state representation model.The key to the basic idea is to use the highly connected structure of tensor and the good solution of the optimization technique,and obtain the core test set directly from the original tensor.First,consider how to build and solve the optimization model for learning its PSR model,then give the model parameters of its PSR model,and finally extend it to the case of multi-agent.Therefore,from the solution of the model,the core joint test set and model prediction parameters of the PSR model can be directly obtained,and then some auxiliary matrix can be constructed to obtain the model transfer parameters.