Research On Online Network Resource Allocation Method Leveraging Multi-agent Deep Reinforcement Learning In Edge Network

With the rapid development of information technology,a large quantity of computingintensive and delay-sensitive emerging applications(such as augmented reality(AR),virtual reality(VR),cloud games,etc.)have emerged,bringing users unprecedented experiences.However,these applications are largely limited by the constrained resources of user mobile equipment.Relying on the booming cloud computing technology in the past few years,users can upload the complex computing tasks involved in these applications to powerful cloud servers for processing.Although this method can solve the problem of limited resources of user equipment,since cloud computing servers are often deployed far away from users,it may cause a large transmission delay,which cannot meet the requirement of delay-sensitive applications such as augmented reality.Mobile Edge Computing(MEC)deploys computing resources in an edge network close to the user side,enabling users to offload complex computing tasks to edge servers.Compared with cloud computing,mobile edge computing introduces lower transmission delay and can provide much more powerful computing resources than user equipment.However,various resources(including computing resources,storage resources,and bandwidth resources)in edge networks are relatively limited compared to cloud networks.As the number of user devices in edge networks continues to increase,it is common for users to compete with each other for edge network resources.Therefore,it is necessary to reasonably allocate edge network resources to improve user experience quality and resource utilization.However,traditional network resource allocation methods are difficult to adapt to the highly dynamic and online decision-making edge network environment.Multi-Agent Deep Reinforcement Learning(MADRL)method,as an emerging online decision-making method of multiple decisionmaking agents,has been widely studied in both academia and industry in recent years.By modeling the decision problem of multiple agents as a Decentralized Partially Observable Markov Decision Process(Dec-POMDP),MADRL methods could train each agent's policy to optimize the system objective.Obviously,the MADRL paradigm is very suitable for solving the resource allocation problem of edge networks that contains multiple decisionmaking agents and requires online decision-making.In this thesis,the MADRL algorithms based on Centralized Training and Decentralized Execution(CTDE)framework are employed to solve three typical resource allocation problems in edge networks,and their performance is verified by experiments.In the problem of adaptive bitrate algorithm for clients in the scene with multiple DASH traffics,this thesis obtains an adaptive bitrate scheme for clients that compete with each other for the bandwidth of a bottleneck link based on the counterfactual multi-agent(COMA)policy gradient.This scheme achieves high performace for quality of experience while maintaining fairness among video clients.In the problem of joint cache replacement strategy design for multiple basestations in edge network,also by using COMA policy gradient,this thesis obtains cache replacement policy for each basestation in edge network.By employing the obtained policy,basestations could cooperate with each other and achieve a good trade-off between system cost and content hit rate.In the problem of designing edge server side resource allocation and scheduling method for augmented reality business,this thesis models the decision process of parameters for processing the image requests into a Dec-POMDP by employing action space decomposition method.The designed scheme is based on Multi-Agent Deep Deterministic Policy Gradient(MADDPG)and achieves high performace of Qo E while maintaining fairness among AR clients.