Design And Implementation Of Self-adjusting Dynamic Stream Processing Engine

Traditional batch processing system and stream processing system are mainly aimed at real-time processing of large-scale data.The development and deployment of the system is complicated,and the reconstruction and use of data processing applications are difficult.With the development of application performance management,there is an urgent need for a lightweight,stream-processing engine with good reconstruction capabilities to meet the streaming processing needs of multi-source structured log data required for application performance management.Based on the data processing requirements of application performance management,this paper analyzes the existing problems of the existing stream processing system in the cluster node dynamic management,message persistence,message recovery,task execution and other aspects,and proposes a dynamic adaptive streaming data Approach.This method solves the traditional flow data processing engine such as Kafka,which is easy to appear when dealing with the application performance management data analysis system by introducing ZeroMQ-based cluster peer-to-peer broadcast strategy and the time window algorithm based on Redis-based AOF persistence feature.The problems of inaccurate synchronization of cluster nodes,slow message recovery,and overly complex cluster configuration improve the reliability of node data processing and the high efficiency of cluster management.The paper first analyzes the cluster node management requirements and node fault recovery requirements of the application performance management system,and designs a dynamic adaptive streaming data processing engine that includes a distributed processing engine and a stream processing node proxy.The distributed processing engine realizes the dynamic discovery and management of cluster nodes by encapsulating peer nodes broadcast in the ZeroMQ cluster,and the data messaging between nodes.Stream processing node agents encapsulate data acquisition and distribution through the "producer-consumer" model to decouple stream data processing nodes and perform dynamic management of data processing.The stream processing node adopts the time window algorithm to process the data self-adaptive loading of the node during the fault recovery process and complete the adaptive management of the data processing.After that,the dissertation designs and implements the core functions,developer interfaces,data structures,and data flow of the dynamic adaptive streaming engine,providing the developer with a complete lightweight data transmission and processing service.At last,the paper tests the function and performance of the engine through a typical application performance management application,and verifies the effectiveness of the system.It also summarizes and forecasts the lightweight streaming data processing engine.