The Real-time Capture Analysis System Based On Stream Compute

Because of the close relationship between financial industry and data,the financial industry has high requirements for data processing security,stability and real-time.The huge,persistent data flow in the internal network of the financial industry has its own unique value after the real-time analysis and processing.Comparing with the large data source for the geometric growth of the computing speed,the processing of mass online flow data is more difficult because of the strict requirements for real-time.This paper mainly focused on how to deal with the mass flow data of financial industry and the construction of flow data capture and establishment of flow data capture analysis architecture which was based on distributed processing mechanism.On the basis of the function of data capture and analysis,the system of related services was built.At the same time,it also realizes the data API service,data push and network situation monitoring.First,a distributed streaming data processing system which was applied to the processing of MBPS(Million Bits Per Second)level data was designed based on network data acquisition.A distributed computing system based on network is established.Based on Pcap network data acquisition and distributed computing system,this system completes the high-speed real-time resolution of data through the use of multi-network collaboration.At the same time,two common distributed frameworks were compared and advantages of frameworks were summarized and selected.Based on the experimental result and the mechanism of the frameworks,a storage framework and a flow model for high speed data were established.Secondly,based on data came from different business systems,a universal protocol analysis method is proposed for different generic protocols.According to the static/dynamic protocol semantics and syntax format in,we have made corresponding protocol templates which were written as rules for different protocols.Then,by using the loading template file,the corresponding data structure is constructed.Then parsing of the protocol was completed through iterative updating and parsing.Finally,a series of services was established based on Distributed processing framework system services and included the queue service,file server,database service,data service,warning push from the data collection and processing to the output.And we finally complete the real-time data traffic analysis monitoring and alarm in order to support data acquisition,data transmission and data processing overall seamless connectivity requirements.