Deep Reinforcement Learning Based Vertical Handoff Algorithm For Heterogeneous Wireless Networks

The next generation wireless network will be coexistence of a variety of radio access networks,including cellular networks,wireless personal area networks,wireless local area networks and wireless metropolitan area networks,etc.To solve the contradiction between the scarcity of the radio resources and the rapidly increasing number of mobile users calling for high quality of service,different wireless communication systems have to be built up together to form heterogeneous wireless networks.However,the differences in coverage,transmission rate and access technology of different networks may give rise to barriers to the integration of heterogeneous wireless networks.Vertical handoff is one of the technologies to solve the problems mentioned above.Vertical handoff plays an important role in supporting the service continuity and guaranteeing the quality of service,which has important investigation significance.The core of vertical handoff technology in heterogeneous wireless networks is to determine when to switch users to an appropriate candidate network.Therefore,“when” and “appropriate” are the focus of this paper.In this regard,this paper proposes two intelligent handoff algorithms based on deep reinforcement learning from different focuses.Firstly,a vertical handoff algorithm based on Deep QLearning Network(DQN)is proposed,which transforms the vertical handoff decision into reinforcement learning problem.The analytical hierarchy process(AHP)is used to calculate the weights of different Qo S parameters,and the reward functions of real-time services and non-real-time services are constructed respectively to calculate the immediate reward value.On this basis,the back propagation(BP)neural network is used to approximate the value function in reinforcement learning,and the initial parameters of BP network are set with the evolution strategy(ES)with global optimization capability in order to improve the convergence speed and precision of parameter learning.A training database is constructed to store the historical handoff decision data,which is used as a sample for the training of neural network.The parameters of BP network are trained by gradient descent method,and the handoff strategy is updated with the goal of maximizing the expected value of cumulative discount reward.The simulation results show that the ES-DQN algorithm can select the most suitable wireless network for users of different service types.The proposed method not only outperforms existing schemes with handoff latency and throughput,but also reduce handoff failure probability and new call blocking probability.A vertical handoff algorithm based on Deep Deterministic Policy Gradient(DDPG)is proposed to solve the problems of frequent handoff,excessive terminal energy consumption and waste of network resources caused by fixed decision intervals in fast-moving terminal scenarios.The mobile characteristics of the terminals are incorporated into the construction of vertical handoff algorithm,and the vertical handoff process is divided into two sub-processes: handoff time selection and network selection.In the sub-process of handoff time selection,DDPG method is used to adjust the decision time interval adaptively according to the mobile speed of the terminal,so as to avoid frequent handoff of fast-moving users in heterogeneous networks and improve the handoff efficiency.In the sub-process of network selection,ES-DQN method is used for network selection.The results show that the DDPGDQN algorithm can significantly reduce the average handoff times,the blocking probability of new call users and the handoff failure probability,and thereby prevent the Ping-Pong effect.