Research On LTE Radio Resource Allocation Algorithm Based On Deep Reinforcement Learning

The shortage of wireless spectrum resources and its low utilization rate have always been the bottleneck of mobile communication development.As an important part of mobile communication technology,LTE system is also facing the same problem.Studying how to allocate spectrum resources autonomously,intelligently and efficiently is of great significance for improving the utilization rate of communication system resources.Reinforcement learning,as a branch of machine learning,solves the problem of maximizing returns during the interaction between agents and the environment.Deep reinforcement learning,which combines reinforcement learning and deep learning,can handle more complex problems.This thesis applies the deep reinforcement learning algorithm to the downlink radio resource allocation problem of the LTE mobile communication system,and completes the following tasks:Firstly,based on the NS3-Gym open source framework,a simulation platform for radio resource allocation algorithm of LTE mobile communication system is implemented.The NS3-Gym framework is composed of NS-3 and Open AI Gym.In NS-3,C ++ is used to build the LTE mobile communication scenario,which can implement LTE system-level simulation.In Open AI Gym,Python is used to implement the agent part of reinforcement learning.This thesis designs and simulates the situation that the end users are uniformly and non-uniformly distributed among the cells,and realizes that the end user's communication service arrival is subject to Poisson distribution,and the packet size of a single transmission is subject to exponential distribution.A resource scheduler suitable for the resource allocation algorithm proposed in this thesis is designed and implemented.It can synchronously collect user traffic arrival status and channel quality status information in each cell,and can execute a resource allocation scheme input from the outside.The performance evaluation module is designed and implemented,and the problem of unsynchronized reward information is solved by constructing a reward information cache queue.By implementing the interface functions provided by the NS3-Gym framework,it enables information exchange between the reinforcement learning agent and the LTE mobile communication environment.Subsequently,Aiming at the problem of low resource utilization and lack of flexibility in the traditional fixed resource allocation method,a resource allocation algorithm DDPG-RA based on deep deterministic policy gradient is proposed,which models the resource allocation problem as a continuous reinforcement learning task and a transmission time interval TTI in the LTE system as a step of reinforcement learning.The agent observes information such as user service requests and channel quality and gives a resource allocation plan according to its own strategy,then optimizes the strategy based on the feedback of the communication environment.The state,action,reward,value function and neural network structure involved in the algorithm are designed separately.The problem of excessive action space is solved by the continuous representation of the action space.A dual experience pool and weighted experience playback mechanism is designed to solve the problem of sparse rewards caused by the randomness of communication services.Python is used to implement modules such as Actor,Critic and Experience Pool in Open AI Gym.And the information cache queue similar to the performance evaluation module in NS-3 side is designed to ensure the synchronization of state,action and reward information.Finally,simulation experiments prove that the proposed algorithm can optimize the resource allocation strategy,has higher spectral efficiency,and can achieve 91.7% and 94% of the performance of the partial frequency reuse Max C/I algorithm and the full frequency reuse Max C/I algorithm when the system service load is light and heavy,respectively.The comparison between the two scenarios of user uniform distribution and non-uniform distribution shows that the DDPG-RA algorithm can allocate spectrum resources more flexibly.