A Distributed Complex Event Processing Systembased On Publish Subscribe

Complex Event Processing(CEP)is a real-time data processing technology and is suitable for event analysis.CEP identifies event patterns to find events of users' interest and aggregates multiple events according to the correlation of events and the defined rules.CEP creates complex events which suggests more complicated situations.CEP plays an important role in the field of Internet of Things and event-driven systems.Analyzing events generated by sensors and various components of the system will satisfy user requirements better since we can extract more useful information.Most existing CEP engines are centralized architecture and are unable to meet the demand of handling large-scale of events due to the increasing number of events.With the advent of distributed technologies,there has been more and more distributed frameworks,making it possible to design a CEP system through distributed architecture.Combining distributed technologies with CEP can provide users more powerful functions.There are already some researches on distributed complex event processing system,however,they didn't focus on the scalability and fault tolerance of the system.This paper proposes a distributed complex event processing system based on publish subscribe.We design a CEP model of Directed Acyclic Graph(DAG)and a communication mechanism based on pub/sub.The loose coupling and asynchronous of pub/sub make it easy to do parallel processing.The paper's work is as following:Firstly,we propose a CEP model based on DAG.In this model,the vertices of the graph represent CEP operations.The edges of the graph represent the data flow between the CEP engines.This model describes the hierarchical relationship between the events visually.Secondly,we propose a communication mechanism based on pub/sub.The mechanism maps the input-output relationship expressed by the edges in the DAG to publish-subscribe relationship.As the result,CEP tasks of the same level will work in parallel and the system is easily to be scaled up.Thirdly,we design and implement a distributed complex event processing system.The resource management component of the system is responsible for allocating resources for CEP applications submitted by users and controlling the lifecycle of the applications.Moreover,it has good fault tolerance and scalability mechanism.The CEP engines deployed in the nodes of the cluster communicate with each other through the pub/sub method and provide the function of executing CEP tasks.Finally,the experimental results show that the CEP flow model and pub/sub mechanism is effective and our proposal is scalable and fault-tolerant.The system achieves the goal of complex event processing and the performance reaches user needs.