Attention-Weighted Federated Deep Rein-Forcement Learning For Device-to-Device Assisted Heterogeneous Collaborative Edge Caching

Currently,the wide utilization of Internet of Things(Io T)has brought rocketincreasing service requirements for existing mobile networks(i.e.,4G),which results in the rapid growth of mobile data traffic.Advances in sensing and artificial intelligence(AI)techniques have enabled innovative intelligent applications for improving people’s daily life.However,these applications are highly dependent on the computation,storage and communication resources.The low latency for content access and diverse application requirements may not be satisfied if the contents are fetched from remote data centers(e.g.,cloud server).To address these issues,it is necessary to introduce advanced networking architectures and new data transmission techniques towards next-generation mobile networks and beyond(i.e.,5G/B5G).Particularly,mobile edge caching(MEC)has been regarded as a promising technique to relieve the burden of backhaul traffic for network operators.In the MEC system,popular contents can be cached in proximity to the edges of networks,e.g.base stations(BSs)and user equipment(UE)(or mobile devices),which reduces massive duplicated traffic of content deliveries via backhaul networks and shortens the transmission delay.Meanwhile,by combining device-to-device communications,the network performances on traffic offloading and delay reduction can be further improved.There have been some researches on edge caching,but most of the existing research solutions on edge caching are based on traditional caching algorithms,which cannot adapt to dynamic network environments;some researchers also propose methods based on machine learning to make complex edge caching decision.However,in these algorithms,it is challenging to train the AI model.Sending the local data to the cloud server will waste massive communication resources and also risk users’ privacy.In order to solve the above challenges,in this thesis,based on the flexible trilateral cooperation among user equipments,edge base stations and a cloud server,we propose a D2 D assisted heterogeneous collaborative edge caching framework by jointly optimizing the node selection and cache replacement in mobile networks.The main contributions in this thesis are 1)We investigate the issue of D2 D assisted heterogeneous collaborative edge caching in mobile networks.Particularly,we model the whole edge caching process as an LT-MILP problem and use the DQN model to control the decision process of joint node selection and cache replacement dynamically based on the network state and historical information.The DQN model can continuously update the local caching strategy based on the interaction with the environment,so compared to some traditional caching algorithms,it can make better dynamic caching decisions.2)We propose the AWDFRL framework,an improved FL framework which can train the DQN model in a distributed manner through keeping the data in the local UEs and address the issue of model aggregation among heterogeneous UEs.Most importantly,we employ an attention mechanism to control the model weights in the FL aggregation step,which can solve the imbalance problem of local model quality.In addition,we derive the expectation convergence of AWFDRL.3)Simulation results show that compared with existing methods,the proposed AWFDRL framework can effectively reduce average delay,improve hit rate,and offload traffic.Moreover,compared with existing federated learning architecture,our architecture can achieve higher training efficiency and greater model reward.In summary,in this thesis,we use DQN(a reinforcement learning method)to manage the cache in collaborative edge caching network.Compared with the traditional caching algorithm,DQN model can capture the content request pattern,so it can achieve higher cache hit rate.To tackle the problem of model training and device heterogeneity,we propose the AWDFRL framework an improved FL framework based on attention mechanism so we can train the local model without sending local data to the remote server.Meanwhile,we have also optimized the aggregation period between heterogeneous devices based on attention weight.