UAV Path Planning Based On Deep Reinforcement Learning Aided By Radio Map

Thanks to the low cost and flexible mobility of UAVs,they have been applied to various complex communication scenarios.The traditional UAV communication trajectory planning is performed by manually modeling the problem mathematically and then solving it optimally,often assuming that the channel model is a simple lineof-sight(Lo S)or probabilistic Lo S,obtaining a functional expression about the UAV position,and then using convex optimization techniques for path planning.Due to the time-varying nature of the channel and the mobility of the UAV,the traditional modeling and description are too simple and the obtained trajectories cannot guide the practical application of the UAV.Deep reinforcement learning is a method for optimal decision making in complex dynamic environments.Optimization of parameters such as UAV trajectories and resources in a dynamic low-altitude network is actually an optimal decision problem in a dynamic environment,which fits with the goal of deep reinforcement learning.Therefore,in this paper,we abandon the traditional modelingbased solution method and use DRL to construct the channel environment of the lowaltitude network,generate radio maps,and then use the radio maps to assist in UAV trajectory optimization.How to use the mobility of UAV to construct radio maps and accomplish communication tasks more efficiently is a problem worth studying.This paper is dedicated to investigate the deep reinforcement learning algorithm based UAV trajectory optimization strategy,and the specific research work is as follows:We consider a terrestrial cellular network-connected UAV communication scenario,where control of the UAV is achieved through a ground base station,and a trajectory is designed for the UAV to achieve minimal mission completion time.To ensure safety,the reliability of the UAV connection to the ground network needs to be ensured as much as possible.And considering the reality that there is a coverage black hole in the coverage of base stations in low altitude,we propose two different metrics for the connection constraint,based on which the UAV’s flight time minimization problem is formulated.First,the duration of a single disruption of the UAV during flight is less than a set threshold.Second,the cumulative time of multiple interruptions of the UAV does not exceed the given threshold.We propose a multistep learning algorithm based on Dueling DDQN,modeling the flight time minimization as a Markov decision problem,rationalizing the basic elements of reinforcement learning: state space,action space,and reward function,and simulation results demonstrate the effectiveness of the designed deep reinforcement learning algorithm,which can well solve the mobile-aware trajectory design problem.Aiming at the scenario of UAV wireless energy supply communication network,where the UAV supplies energy to ground devices while receiving data sent from the base station,we study the problems of maximizing the total energy transmission,maximizing the total data throughput and minimizing the energy consumption of the joint UAV,and propose the UAV flight trajectory design and flight strategy optimization.We use an improved DDPG algorithm to obtain the control strategy of the UAV,transforming the dynamic wireless functional network problem into a Markov sequence decision problem and setting a multidimensional reward function corresponding to the three optimization objectives mentioned above.Numerical results verify the effectiveness of the DDPG algorithm,which can optimize the path to supply more energy and receive more data simultaneously with less energy consumption.The DDPG algorithm is able to obtain the optimal strategy compared with the proposed two benchmark solutions of maximum speed and most energy efficient speed.