Distributed Complex Event Stream Processing Engine Research

With the development of cloud computing, many researchers focus on big data recently. However, there are still a lot of challenges, such as dealing with semistructured data like XML, the representation and conversion of data, efficiency of processing, how to suit develop environment of different applications, etc.Based on event-driven architecture, Complex Event Processing(CEP) is one of the key technologies of big data processing, which combines simple events, event stream processing and complex events for processing. CEP can extract event sequences with specific patterns for real-time processing to meet the demand of high throughput, low latency, etc. in big data processing. Currently, a large number of CEP languages and stream computing platforms have been proposed, but they still have limitations.Since existing several problems in CEP engine, we propose a distributed complex event stream processing engine based on CEStream(a CEP language), providing an event detection function based on regular tree pattern, which can both support regular expression matching of time series and structural constraint of hierarchical data. The engine can also capture data from multiple sources and detect the composite event which supports regular expression pattern with time series. According to the detection demand of multiple sources composite event, we proprose a method of pattern decomposition. The method can decompose the task of complex event processing into several independent event detection tasks and deploy the independent tasks on different nodes of cluster and remote event detection agents. The transfer consumption of single source data can be reduced through the agents, and the efficiency of multiple sources event detection is improved by using parallel computing of cluster.The experiment proved that the system achieved query function of CEStream language and the particular features such as regular tree pattern matching and composite event detection of multiple sources. Furthermore, the detection efficiency of system was actually improved by the decomposition method. The system reached the design target of low delay and high throughput which can satisfy the mainstream requirements of complex event stream processing.