Research On Multi-Agent Area Coverage Based On Reinforcement Learning

Focused on area coverage problem in air-ground multi-agent systems, a decentralized partially observable Markov decision processes (DEC-POMDPs) is established to describe a heterogeneous multi-agent system, and reinforcement learning algorithms are applied to solve decision making problems in multi-agent system. Major topics addressed in the presented thesis include:First of all, the developing process from MDP to DEC-POMDPs is introduced. When partially observable feature and uncertain sensing feature are considered, agents’observations are no longer Markovian, which result in an NEXP-complete problem on optimal solution of DEC-POMDPs.Secondly, an area coverage simulation environment based on POMDP model is established. QMDP and Q-learning algorithms are used to help making action decisions. The influence of different observation uncertainty and observation accuracy is analyzed by adjusting parameters of the POMDP models.Thirdly, an online planning algorithm based on DEC-POMDPs is applied and verified in isomorphic multi-agent simulation scenarios and air-ground heterogeneous multi-agent simulation scenarios. Considering low time consumption and high timeliness of communication action, a heterogeneous multi-agent reinforcement learning framework is designed, which contains a plurality of reinforcement learning modules. By reducing the communication frequency under the premise of not affecting convergence rate distributed reinforcement learning algorithm, this framework improve the ability to identify state and the efficiency of decision-making in hidden Markov model.In order to effectively assess and analyze the performance of the above mentioned algorithms, a multi-agent reinforcement learning toolbox based on MATLAB software is developed. In this toolbox, a sophisticated reinforcement learning simulation framework is established, which includes kinematic model module, environment and map module, reinforcement learning module and etc. A persistence layer module is developed to improve the ability to deal with big data of MATLAB software, as a computational support for verifications of the algorithms. Coupled modes of modules are optimized by predefining interfaces in each modules and object-oriented programming methods, which make this toolbox easy to use, improve and secondary develop.