Research On Cell Association Algorithm Based On Deep Reinforcement Learning In Ultra Dense Networks

With the rapid development of mobile internet technology,the number of terminal devices shows a geometric growth trend.At the same time,the types of services are increasingly rich,and the requirements for data traffic and delay are also increasing.In order to meet the data traffic demand of future wireless communication,the network will tend to be compact.Ultra-dense networks are the dense deployment of small cells in the communication scenes,each terminal may be covered by multiple cells,so it will produce complex association problems.This dissertation focuses on the cell association problem in ultra-dense networks,that is,how to associate the terminal with the cell.The goal is to take into account the system performance of the network and the time cost,and quickly realize the cell association decision.Firstly,a cell association method based on Q-learning is proposed.Considering that the time cost of this method is still relatively high,a method based on deep Q-learning is proposed by combining the decision-making ability of Q-learning with the prediction ability of deep learning.This method is divided into two stages: training and application.In the training stage,the deep Q-network is trained offline through the interaction between agent and environment.In the application stage,the trained neural network is used to adaptively obtain the optimal action selection strategy,that is,the final cell association result is obtained.According to the simulation results,when the number of users in the scene reaches 10000,the time cost can still be maintained at millisecond level.Further considering the dynamic time-varying of communication scenarios,from a macro point of view,the real life scenarios are always changing,and the user distribution characteristics in the scenarios will not remain unchanged,but are changing dynamically,therefore,the obtained system performance of the cell association results by using the same trained neural network in the constantly changing scenarios will have a greater loss.So,according to the correlation of adjacent scenes,we propose a fast cell association algorithm which can ensure the system performance loss is less than the former at the same time.According to the simulation results,this method can complete the cell association decision with better system performance in a short time.The above methods are based on the method of single-agent learning.Nowadays,multi-agent learning has been widely used in many fields.Therefore,we also try to solve the cell association problem in ultra-dense network by using multi-agent learning.A cell association method based on multi-agent reinforcement learning is proposed.Each user is regarded as an agent,and the result of cell association is obtained by updating their Q table through interaction with the environment.Further,the multi-agent deep reinforcement learning method is proposed,and each user maintains their own deep Q-network to guide cell association.The simulation results show that we can solve the problem by using multi-agent learning method.The performance and algorithm time of the system are basically the same as that of the single-agent learning method.