Predictable Monitoring In Distributed Computing

In the past few decades, large-scale distributed computing systems have been widely used to serve a growing number of applications. Lots of firms have deployed their own large and complex distributed systems consisting of thousands of computers. In the distributed computing system, resources are shared by large amount of users in a time-shared manner. In such environment, resources should be strictly assigned to the applications to achieve high performance. Thus, resource monitoring and usage prediction is required for the scheduling. Among these resources, CPU load plays an important role. So in this paper, we focus on the CPU load prediction.In recent years, some research in the field of CPU load prediction has been carried out. Many prediction models were developed, such as Network Weather Service, one of the most popular performance monitoring systems. However, most of these models are one-step-ahead or short-term predictors, whose prediction range is too short to satisfy the requirements of the scheduler. There are also some long-term prediction models. But these models have their own drawbacks, such as low prediction accuracy or only suitable for specified type of CPU load, etc.In order to solve these problems, we propose two different long-term prediction models, tendency-based prediction model and pattern-based prediction model. The tendency-based preidiction assumes that the next few values will change according to its historical variation tendency. It applies Fourier transform to exploit the periods of the CPU load waves, and uses many tendency-based methods to perform prediction. The pattern-based prediction matches the current pattern with the historical data to find out the similar pattern, and predictes the next few values based on the data following the similar pattern. Many experiments were carried out to evaluate the prediction accuracy of the above two models. The result illustrates that these two models can effectively play short-term and long-term prediction within certain fault tolerance.Then, we present a semantic event based distributed monitoring system, which wrappes its collected data into the predefined event type. Users can define and register their own event type, so this monitoring system has splendid scalability and can be applied to different resources. Besides, this distributed monitoring system also integrates many prediction models into its self-learning module for intelligent management. Through many tests, we can conclude that the semantic event based distributed monitoring system works well for monitoring and prediction.