Research And Implementation Of Data Stream Processing Technology For Environment Monitoring

With explosive applications of information processing in communication, industry manufacturing, business processing and many other areas, more and more data take the form of continuous data streams rather than the traditional finite stored data set. The appearance of continuous data stream models is bringing great challenges to traditional data management technology. The existing database management techniques have hardly been applied to process data streams effectively. Therefore, it's necessary to study the new data stream management techniques according to the characteristic of data stream models. Nowadays the study on data stream management is becoming one of the hot topics among the database circle all over the world. To study data streams has not only important scientific value but also huge application outlooks.The large-scale integrated management information system for material requires the temperature sensors, humidity sensors, smog sensors and other sensor equipments to monitor the environment. These sensor data are reflected in the form of the dynamic and continuous data stream. In light of these inquiries that sensor data processing and management need, we focus on the join aggregation continuous queries on the continuous data streams in this paper. Firstly, to deal with the problem of low efficiency when the traditional algorithm named J-A processing over data stream encounters, an improved algorithm named J&A~+ is proposed, which computes the aggregate function using the last aggregation results and the changing data of sliding window between two executions of query. The J&A~+ algorithm can efficiently give the accurate query result under the condition of consuming less memory. Secondly, based on the J&A~+ algorithm, a new data stream real-time processing strategy is proposed in this paper. It not only improves the real-time data processing efficiency, but also guarantees that the monitoring system captures the exceptional situation efficiently and effectively. Finally, according to the theory mentioned above, one of the integrated management information system for material named environment monitoring system based on the data stream technology is designed and implemented. It solves the problem that the environment monitoring system based on the traditional data processing technology processing over data stream encounters. When the environment monitoring system based on the traditional data processing technology implements the streaming data queries, it needs a large number of I/O switching which leads to the issues of inefficiency and can hardly meet the need of real-time systems. Recently the environment monitoring system has a successful trial in the integrated management information system for material.