Research On Multi-dimensional Resource Allocation Methods For Wireless Heterogeneous Networks Based On Deep Reinforcement Learning

Heterogeneous networks have become an important deployment method for the 5th Generation Mobile Communication(5G).It can effectively improve the spectrum utilization and network capacity by deploying various types of small base stations within the coverage of macro base stations to shorten the distance between terminal equipment and base stations.However,the commercial era of 5G has driven the rapid growth of data traffic and smart devices,while spawning new smart applications with diverse service needs.To meet the growing service demands in the future dense heterogeneous networks architecture and alleviate the shortage of network spectrum,computing and caching resources,many new technologies such as Device-to-Device(D2D)communication,millimeter wave,Mobile Edge Computing(MEC)are widely used,which not only improves network performance but also brings new challenges to wireless networks resource allocation.On the one hand,the diversified service requirements promote the expansion of resource allocation problems from a single communication resource to the joint optimization of communication,computing and caching resources.The multi-dimensional resources are closely linked and coupled with each other,which increases the difficulty of multi-dimensional resource allocation.On the other hand,the heterogeneous network environment is dynamically complex,the environment information is difficult to obtain accurately,and the variable dimension in the optimization problem is high,so it is difficult for traditional optimization methods to quickly solve the multi-dimensional resource allocation problem in the complex dynamic environment.In recent years,artificial intelligence,especially the Deep Reinforcement Learning(DRL)method,provides an effective way to solve the complex optimization problem in wireless communication networks resource management through the dynamic interaction optimization strategy between the agent and the environment.Therefore,this thesis focuses on the DRL-based multi-dimensional resource allocation methods of communication,computing and caching in wireless heterogeneous networks.The main research contents are as follows:1)To address the challenges of spectrum sharing and interference management among user devices,this thesis studies a D2 D communication mode selection and channel allocation method for heterogeneous networks based on Deep Q Network(DQN).First,a heterogeneous network with cellular and millimeter-wave frequency bands that supports D2 D communication is proposed,where D2 D users can choose cellular mode or millimeter-wave mode for communication.Second,the joint optimization problem of D2 D communication mode selection and channel allocation is formulated as maximizing system sum rate while guaranteeing minimum quality of service requirements of both cellular and D2 D users.Finally,a resource allocation algorithm based on distributed multi-agent DQN is designed to optimize the mode selection and channel allocation strategy.Simulation results show that the proposed algorithm has good convergence,and can achieve better system performance compared with other existing schemes.2)To meet the low-latency requirements of content request services and alleviate the pressure of limited server cache capacity,this thesie studies a cooperative edge caching method in heterogeneous networks based on Multi-Agent Deep Deterministic Policy Gradient(MADDPG).First,a two-phase cooperative edge caching strategy including content placement phase and content delivery phase is proposed.Second,the joint optimization problem of content caching and bandwidth allocation is formulated as minimizing the difference between content delivery delay and cache hit rate in each small cell.Finally,a MADDPG-based cooperative edge caching algorithm is designed to solve the optimization problem,which realizes the intelligent decision of content caching and bandwidth allocation.Experimental results show that the proposed algorithm can effectively reduce the content delivery delay and improve the cache hit rate.3)To solve the problems of diversified service requirements and the limited computing resources and cache capacity of MEC servers,this thesis studies a multi-dimensional resource allocation method in heterogeneous MEC networks based on MADDPG.First,a heterogeneous MEC network architecture with different service types is constructed.Then,partial computation offloading and cooperative edge caching strategies are proposed,and the communication,computing,and content delivery models are introduced in detail.Second,a joint optimization problem of multi-dimensional resources is formulated to minimize the cost of each small cell.Finally,a multi-dimensional resource allocation algorithm based on MADDPG is designed to effectively optimize computation offloading,content caching and resource allocation strategies through centralized training and distributed execution.Simulation results show that the proposed algorithm can effectively reduce the cost of small cells and obtain better system performance.