Analyze And Predict The Flash Crowd On Storage System

Flash crowd means lots of accesses suddenly happen to web server in a short time, while the web server cannot handle the rapid rise accesses and is unable to complete the users’ requests smoothly. As the traditional computer system often configures the resources according to the expected peak load statically, the flash crowd may result in the energy consumption, waste of resources and so on. However, due to the randomness of flash crowd, it is always a challenge to predict the flash crowd effectively. This paper uses a data mining method to get the frequent correlation I/O rules from storage system, and then filter out the rules to predict the flash crowd.In order to improve the efficiency of mining the frequent correlation I/O, we introduce the strengthen correlation window which takes the Apriori algorithm as basis and makes consideration of the time properties of I/O data. While in the process of generating the frequent correlation I/O, the window can reduce time overhead, achieve awareness of temporal locality and enhance the correlation of rules with the default size and uncertain step. Experiment results show that the proposed algorithm can acquire the consistent results as the Apriori algorithm does. Moreover, the new algorithm can reduce the time overhead by 20% to 40%.In addition, we finds that the factor which leads to the flash crowd is not the frequent correlation I/O. According to this finding, we propose a burst prediction algorithm based on association mining. The algorithm mainly consists of five measures. Firstly, producing the frequent correlation I/O rules by the training dataset of three real I/O traces. Secondly, dividing the dataset in accordance with a longer time granularity and getting the corresponding I/O requests. Thirdly, filtering out the rules which appear in a shorter time granularity. Fourthly, utilizing the block I/O which is filtered out the rules to predict the flash crowd in a longer time granularity. Finally, figuring out the hit rate and incidence of the predictive method. After making sure the burst threshold and observation threshold, we complete the verified experiments with the test dataset of three I/O traces, and compare with the traditional time series prediction algorithm. Our experimental results show that this approach can get about 70% hit rate in the ideal case, which is twice times higher than the time series prediction. Consequently, it is feasible and practical to predict the flash crowd in a longer time granularity through the non-frequent correlation I/O in a shorter time granularity.