Web Traffic Characteristics Model Research And Application

Web caching technology is an effective approach to improving Web performance. The management of Web caching is the key issue of Web caching technology, and the study of mathematical model of Web reference characteristics is the basis of effective management of Web caching. By the study of the reference behavior of Web users and the reference characteristics of Web object profoundly, some common characteristics identified in proxy workloads can be found: (1) the Web object popularity often follows Zipf's law; (2) the file size distribution for Web documents follows heavy-tailed; (3) reference locality characteristics exists in the Web proxy reference stream.Locality characteristics of Web reference streams have great impact on performance and design of caching and prefetching system. Locality characteristics of Web reference streams includes the temporal locality and the spatial locality. Using mathematical model method to study the locality principle plays an important role on the design and performance improving of caching and prefetching system. Through analyzing and discussion, a new method to model Web workload locality characteristics is established based on the study of entropy and coefficient of variation. Entropy can reflect the temporal locality better than Zipf's law, and the coefficient of variation can be used to solve the problem of modeling of spatial locality between the same files. Experiments and discussions show that these metrics can depict locality characteristics more properly and are much easier to use.The Web proxy workload generator can generate synthetic workload in terms of Web reference characteristics appointed by users, which provides the theoretical basis on studying Web cache systems. A synthetic Web workload generator (WebSimGen) is designed, implemented and validated in terms of mathematic model established by the Web characteristics in this paper. Based on a two-level caching proxy structure and the synthetic Web workload traces, the simulated experiments make use of ADF (Aggregation Disaggregation and filtering) model to evaluate the reference characteristics and performance of synthetic Web workload. The experimental results have shown that the synthetic traces have the similar performance characteristics with the real ones. Web cache replacement algorithms (LRU, LFU, GDSF) are also evaluated carefully using synthetic workload generated by WebSimGen and real workload. The experiment shows that in the two-level caching hierarchy if the lower level proxies use the LRU or the LFU policy, the GDSF policy is the best policy at the upper level. The generator has great flexibility, which can provide an important basis for further improving Web caching performance and the study of prefetching technology.