Research On Caching Strategy Based On Reinforcement Learning

In recent years,wireless data traffic has been increasing dramatically,which causes the wireless network infrastructure difficult to handle such huge data,especially during peak communications.Wireless caching technology has been proposed to alleviate the traffic load,where the basic idea is to pre-cache the most popular files into the wireless node during off-peak time.Take a cache-aided cellular network as an example,if the requested file is stored in the small base station(SBS),the SBS can directly send the file to the user,thereby reducing the backhaul link load.For the wireless caching,the most important problem is how to devise the caching placement strategy,i.e.,pre-store what kind of content to what kind of wireless nodes in the networks.Regarding this problem,our main research is as follows.First,we investigate the background of wireless caching technology,common caching strategies,file popularity settings and performance evaluation criteria,and then investigate the basic concepts and classical algorithms of single-agent and multi-agent reinforcement learning.Secondly,we investigate the probabilistic caching placement in a heterogeneous cellular network,where there are several types of base station,and each type of base station has a different cache capacity.We measure the network transmission performance by introducing the average service success probability.Taking into account stochastic geometry,Zipf law and the signal-to-noise ratio coverage model,we provide the cache hit probability and the successful transmission probability to obtain the average service success probability.The system is optimized by maximizing the average service success probability,through optimizing the probability caching placement.As this problem is non-convex,and we turn to use heuristic algorithm to solve the placement strategy.Simulation results are provided to demonstrate that the proposed placement strategy is superior to the conventional most popular content(MPC)caching strategy.Finally,we investigate a distributed caching strategy based on multi-agent reinforcement learning in a cache-aided network.The wireless nodes can collaboratively optimize distributed caching strategy to maximize network performance measured by the average cache hit probability.Specifically,we firstly model the distributed caching strategy problem as a fully cooperative repeated game and then analyze how to improve the average cache hit probability under the multi-agent reinforcement learning framework.We further propose the caching strategy based on the Frequency Maximum Q-value(FMQ)and the caching strategy based on the Distributed Q-learning(DQ)to optimize the distributed caching strategy.Simulation results show that the proposed FMQ-based strategy significantly improves the average cache hit probability,while the proposed DQ-based strategy can converge to the optimal strategy with probability one.Moreover,the proposed FMQ-based and DQ-based strategies are not only superior to Q-learning based strategy,but also superior to probabilistic caching placement and MPC strategies.