Research On Reinforcement Learning Based Routing Protocol For Flying Ad Hoc Networks

With the rapid development of communication technology and the continuous maturity of UAV technology,Flying Ad-Hoc Network(FANET)has gradually become a hot topic in the field of computer network research in recent years.There are many characteristics of Flying Ad-Hoc Network,such as the wide range of UAV nodes,frequent changes of network topology,unstable communication links and so on.Therefore,the design of an efficient routing protocol is an important research content of FANET.According to the characteristics of Flying Ad-Hoc Network,this paper improves and optimizes the Q-Routing protocol based on reinforcement learning to make it more suitable for the scene of Flying Ad-Hoc Network and improve its performance in Flying Ad-Hoc Network.The main contents of this paper are as follows:Firstly,an adaptive AQ-Routing protocol based on reinforcement learning is proposed.This protocol designs a dynamic discount factor based on Q-Routing.The design of discount factor considers the mobility of nodes,the residual energy of nodes and the availability of communication links,so that nodes can choose the best route according to the change of environment.In addition,in order to avoid routing falling into local optimality,a route exploration strategy based on UCB algorithm is proposed to realize the trade-off between exploration and exploitation to make the route converge to the global optimal.Secondly,aiming at the problem that the maintenance cost of large-scale Flying Ad-Hoc Network is too high,which leads to the sharp decline of network performance,a clustering CQRouting protocol based on differential evolution algorithm is proposed.A fitness function based on mobility,residual energy and distribution of nodes is designed to make the clustering structure more stable.In order to prevent the "premature" phenomenon of the algorithm,an adaptive cross-probability adjustment mechanism is proposed to calculate the optimal clustering scheme.The clustering algorithm of CQ-Routing protocol greatly reduces the cost of routing maintenance and prolongs the network lifetime.Finally,the ns-3 simulation platform is used to simulate Q-Routing,AQ-Routing and CQRouting protocols.Experimental results show that CQ-Routing has better performance in packet delivery rate,throughput,average delay,routing control overhead and lifetime.