Research On The Key Technologies Of Distributed Complex Event Processing

In recent years, data generated in a substantial increase in the Internet, How to deal with these data, mining the information contained in the data, become the focus of attention in the computer field. the conventional data processing method is stored the data into the database firstly, and then process these data. But only a small part of the data is of interest, the price of stored all of these data is very large. And in many areas, such as the stock market, detection of electrical equipment, it have a high requirement to the real-time of data processing. Traditional data mining and processing methods can not meet these requirement. In order to extract the interest and higher-level information from the real-time stream of massive data, the complex event processing system appeared. Complex event processing system is able to detect user-defined complex event patterns, and then respond.There is already a lot of complex event processing systems, each system being applied to different scenarios, and for different problem has been improved, However, these systems generally employ a centralized architecture, With the increasing rate of the current generated by the event, the complexity of events continue to increase, the traditional running on a single node of the processing capability of complex event processing system has encountered a bottleneck.To solve these problems, in this paper, we implemented a distributed complex event processing system, which in order to handle large amounts of data, the diversity of the event, by query rewrite methods and distributed processing, improve the ability of complex event processing, The main contribution of this paper includes the following aspects:(1) A new event model and event pattern language, which is more suitable for query rewrite and distributed implementation.(2) Improve the method of complex event pattern based on detection to graph. And propose an algorithm for query rewriting for making complex event queries more efficient.(3) Design of distributed complex event processing framework, and use distributed stream processing framework Storm realization, greatly improving complex event processing system processing capabilities.(4) Through experiments to verify the validity and efficiency of algorithms and systems.