| Content Delivery Network(CDN)delivers content to users through nearby edge caching nodes.CDN delivered 77% of internet traffic by 2021.On the edge node of CDN,multiple traffic classes often share one cache pool,and their workload patterns are highly complex and strongly dynamic.It is important to dynamically allocate cache space for each traffic class in an on-demand manner to meet the CDNs’ Quality of Service(Qo S).However,the state-of-the-art methods based on Miss Ratio Curve(MRC)modeling that use the "white box" idea for cache allocation are not universal because they are only effective in the scenario using Least Recently Used as cache replacement policy,and these methods cannot be used online due to their excessive computational and memory overhead.To address the problem of cache allocation mentioned above,the cache dynamic allocation problem was modeled as a sequential decision problem using "Black Box" idea,and an end-to-end cache dynamic allocation algorithm(Reinforcement Learning based Cache Allocation,RLCA)based on deep Q-learning was proposed.According to the differences in content size,popularity,workload dynamics,and other characteristics of different traffic classes,a multi-dimensional state space was designed,which can help the model better learn the dynamic workload patterns of different traffic classes on the CDN cache was designed;To eliminate the impact of the performance fluctuation of the workload itself to RLCA model,the short-term and long-term benefits of the current state were defined,and the compound reward function with the best global performance that can tolerate shortterm performance degradation was designed to maximize the overall performance of CDN cache;For the problem of long training time,a discrete action space was designed to reduce the model output dimension and accelerate the convergence;Finally,based on RLCA,a dynamic allocation framework of CDN cache was implemented,which use the two-stage mechanism combining offline training and online usage to reduce the usage overhead of the algorithm.To prove the effectiveness and efficiency of RLCA on dynamic cache allocation in the CDN partitioned cache system,comprehensive performance evaluations were conducted using the real trace of a CDN edge cache node of a well-known domestic cloud service provider.Experimental results show that RLCA can achieve outstanding performance and overhead savings.Compared with the static equalization strategy used in commercial CDNs,RLCA improves the overall object hit ratio by 4.03% and reduces the back-to-source traffic by 7.36%,the average latency by 11.34%.Compared with the state-of-the-art dynamic allocation method based on the miss ratio curve,RLCA reduces the memory overhead by 98.93% and the time overhead by 99.99%. |
