Research On Concurrent Execution Mechanism Of Esper Data Stream Processing Engine

With the rapid development of computer platform and related processing technology,especially the development of data processing in recent years,humanity have entered the era of big data,and the model of processing persistent data has been difficult to meet the demand of many online applications.So a new data mode has emerged in the field of data processing-streaming data.Compared with traditional data,the data elements in the data stream are characterized by infinite number,time continuity,fast perishability and unpredictability.Complex event processing engine can be used to detect the matching data sequence from the streaming data and its performance on real-time processing is excellent.In recent years,it has been more widely used.Esper is a brilliant one of complex event processing engines.Event processing system based on Esper engine can be used for complex event processing and data analysis,which is suitable for dealing with real-time messages and event data flows.Since multi-core processor has become popular and used so common in recent years,so the research of performance with respect to Esper on multi-core processor is very necessary.Esper's official documentation and most of the rest of the resources are mainly about how to uses it,with little analysis of Esper performance analysis and working mechanisms.Recent research shows that Esper has not taken full advantage of multi-core platform computing power.This paper firstly analyzes the working mechanism of Esper engine by analyzing source code.The specific work includes designing test cases,tracking Esper's handling of events in test cases,analyzing key data structures in processing flow,summarizing Esper query logic and event processing logic.Then,the paper analyzes two important lock structures: Managed Read Write Lock and Statement Agent Instance Lock,which are involved in the whole query process of Esper.And it also analyzes the acquisition and release behavior of the two locks and the impact on concurrency mechanism for these behaviors.At last,the working mechanism of Esper's original thread pool model is studied,and the processing flow of concurrent task is analyzed.Based on the aforementioned work,the reason why Esper can not support multi-core platform is also pointed out.Aiming at the problem that Esper can not support multi-core platform,this paper proposes a thread processing framework that supports multi-core platform and implements the framework in Esper's core structure.The new thread framework uses the request queue based on the event type,schedules the worker thread to access the Map-based event storage structure according to a certain scheduling policy and to obtain the event for query processing.In this paper,the implementation of the Java programming language for the thread processing framework is introduced.The specific content includes workflow,scheduling strategy,request processing,and important data structures such as request queue and event storage structure in detail.Finally,in order to verify the performance of the new thread processing framework,this paper compares Esper's original thread pool framework and the framework designed in this paper from many perspectives.Experimental results show that for both the CPU utilization and query throughput,the new framework has a much more increase than the original model.In the best case,the response time can be increased by 20 times.Experimental results further verify that the locking mechanism and query strategy introduced by this paper are the main reasons of Esper's multi-core platform incompatibility.These conclusions can help developers to know better about working mechanism of Esper and develop more effective data stream management system under the multi-core computing platforms.