Research On Query Plan Generation And Opimization In Multi-source Complex Event Detection

With the rapid development of Internet technology,all walks of life are using information technology to judge industry trends and guide the development of the industry.The data generated by various application fields in the process of informationization has also exploded.How to find valuable information from these data has become one of the hot areas of research today.Complex event processing techniques are used to solve such problems.Complex event processing technology is a common means of streaming data queries.With the development of data diversification,the complexity of queries has risen sharply,and its processing performance is facing great challenges.In order to describe complex queries,complex event detection often uses regular expression to represent the relationship between events.The formal type supports Kleene closure operations and has rich description capabilities.In order to improve the efficiency of multi-source complex event query processing,this paper presents a formal query decomposition scheme.Firstly,the complex event processing system is described in the case that the event detection expression ending with Kleene closure operation cannot be decomposed;secondly,the detection expression is decomposed with fine granularity and the corresponding query plan is generated;finally,according to the query Plan to build a cluster topology and run it in a distributed cluster.Under the application background of multiple data sources,this decomposition scheme can effectively improve the utilization of cluster resources and improve the processing efficiency of the system.This paper makes the following optimizations based on the query plan.The analysis of the predicate filtering operations that affect the processing performance is based on the differences in the source of the incidents for the screening,and it is reasonably divided to effectively improve the processing efficiency.For the system that uses the regular tree pattern as the event source matching,the sub-expression that only acts on one event source in the predicate expression can be combined with the regular tree pattern matching to combine the optimization scheme of the design,screening,and integration,which can greatly reduce the system internals.Communication overhead.In the query statement,multiple different regular tree patterns may also be matched for the same event source.For this case,combined with the aforementioned optimization,a common pattern is extracted according to the screening conditions,which can be used for multiple matches at a time,and can improve system performance.This paper designs and implements the CIEP system.The system runs on Storm distributed cluster and carries out experimental tests.It is proved that the above optimization method can improve the query efficiency of the system by about 20%.