The Study On Distributed Complex Event Processing Technology Based On Query Plan

The era of the Internet has produced continuous explosion of data,how to deal with these data to obtain valuable information is the focus of research on computer field.The previous data mining methods need to model the data and design a special algorithm,sensitivity is poor.And these methods can not meet the needs of fast,massive,and diverse data for processing.Complex event processing can process massive amounts of data from different data sources in real time.It extracts meaningful information by combining simple events into high-level events,then to respond.But most of the complex event processing is in accordance with the centralized processing architecture,and now most of the applications,especially for the application of things,the main feature of these applications is the distributed deployment structure.The data generated by the application of the Internet of Things is heterogeneous,decentralized and massive.It is difficult for us to deal with the requirements with traditional complex event processing.In view of the above problems,we mainly study the distributed complex event processing technology,through distributed complex event processing can effectively deal with massive real-time data.The main contents of this paper are as follows:(1)A distributed complex event processing(PDCEP)framework based on event flow partition is proposed,which can receive multiple different data streams and send event flows to complex event processing engine for distributed processing of multiple nodes under the cooperation between the input manager and the query manager,and the results of processing will be sent to the output manager.(2)Based on the PDCEP distributed framework,an operator distributed complex event processing(ODCEP-ESP)method based on event flow partition is proposed.The method includes two aspects:the event partitioning strategy based on the query event type and the distributed method of dividing the complex query.The event partitioning strategy divides the event flow in a planned manner based on the type of event required for the query,each division is treated as a query plan.An operator distributed method is a method of dividing a query into different sub-query sequences to handle complex queries.Each step is assigned to an operator node in the system,which is planned to be processed.(3)In order to solve the problem that there are similar sub-expressions in a large number of complex event query sequences,we propose a method of query rewriting to optimize complex query.Firstly,a corresponding expression graph is constructed according to the query expression and merged(a merge algorithm is proposed in this paper).Then,on this basis,a query rewrite method based on expression graph(ExpG)is proposed.By recognizing the common expression of different expressions,the method is combined,shared and reused to realize filtering redundant queries and improving efficiency.The query expression is represented by a number of corresponding expression graphs,and the expression graph is also a one-to-one relationship with the complex event.Finally,the validity of the algorithm is verified by experiments.