Research On Load Management Technology For Data Stream Analysis System

In recent years, China made great process in the process of information. The basis ofthe process of information is the collection, storage, analysis and utilization of data. Withthe application develops widely and deeply, the widely spread devices of networks declaremore demand on data acquisition size, acquisition precision and acquisition speed. Theamount of data shows the trend of being massive. For the data itself, not only the value isimportant, but also its timestamp. Applications have a higher timeless requirement.Meanwhile, because of variety of acquisition environment and variety of applications, theform and dimension of data is variety.Traditional database has many problems to response to the new features of data.Firstly, the traditional database stores all the data, storage is a bottleneck cause of themassive of data. Secondly, when query data from traditional database, a large number ofIO operations are needed, the timeless of data process is hard to be meted. Thirdly, thetraditional database is unable to adapt to the new demands of stream data management andanalysis.Data stream analysis system(DSAS) is a real-time data acquisition, storage andanalysis system, there are many advantages to response to the challenges brought by thenew features of data. Real-time data stream arrives continuous, fast, unpredictably. Theload management of the system is a big challenge. Storage resources and computingresource are the main bottleneck of the system. Load management mechanism is carriedout based on these two aspects.This paper studies load management technologies, based on the computing resourceand load shedding theory, we design a load management algorithms. We design directedgraph model to describe continuous query in a DSAS firstly. Based on the selectivity andcost of different operators, algorithm estimates the capacity of the DSAS. With the changeof data stream rate, algorithm calculates the load of DSAS to decide whether to shed load.In order to utilize the shake of data stream, we design a method to expect the overloadpoint based on the deadline of queries. The key point of our algorithm is the method todecide where to shed load and the probability model of shedding load.In the experiments, our algorithm reduces Average Deadline Miss Ratio (ADMR) andUtility Loss when the system is severe overload. In the same time, our algorithm has goodadaptability and robust. However, when the system is in mild overload, the method toexpect the point of overload can use idle time followed-up to avoid implement of loadshedding and guarantee the accuracy of queries.