Research On Load Management Technology In Distributed Data Stream Processing

Recently, data stream processing technology has becoming a new research direction in database fields. It wins increasing researchers'attention because of its wide application prospects. Distributed systems, with low cost and powerful processing capability, have inherent advantages in processing fast speed and large volume data streams. Distributed data streams processing technology has been one of the most important parts of data streams research area.In this paper, we study the load management technology in distributed data stream management system, the contributions conclude the following aspects. Analyzes the load manangement technology of existing DSMS prototype system, summarizes the load management technology's characteristics and design principles, improves the Borealis's load management module and establishes a system for experiment. Presents a load balance algorithm based on operator correlation analysis and network traffic analysis, moreover, it reduces the negative impact of operator migration. Investigates load shedding issues on the distributed query network, analyzes the load dependence between each node and improves a distributed load shedding strategy. Studies the existing load shedding technology in sliding window join and proposes a sliding window join query load shedding algorithm which based on output frequency of base window. The study of load management technology in this paper is based on the actual application environment with unnecessary assumptions, provides the theory and application support for further study.