Big Data Flow Processing Analtsis System Based On Kafka

With the development of Internet and information technology,the scale of information systems in enterprises has expanded rapidly,and the complexity of data transmission between systems has become very chaotic with the increase of the number of systems.This may cause the problem of "information island",so that data sharing can not be achieved.This paper by studying the current ETL technology,Kafka message system,Hadoop distributed system architecture,multi-source database technology and combined with Kafka Connect design and implementation of a stream data processing and analysis system.The system supports the extraction,aggregation and distribution of data,implement a large amount of data is sent in and out of the Kafka message system to share with other data sources.This paper mainly introduces the management of clusters,Broker and Topic based on Zookeeper;Kafka Connect constructed different source/sink connectors in upstream and downstream of the Kafka to form a seamlessly channel that enabling the sharing of data among different data sources.At the same time,introduces the monitoring of cluster based on JMX,including the Broker,Topic and connector real-time status monitoring,traffic monitoring and real-time abnormal information early warning;and based on Filebeat and Logstash to implement the log file collection and output in order to query and view the contents of log.Finally,this paper also builds the experimental environment to test the performance of the system.Through the analysis of the results,the system can complete the data from the heterogeneous data source(Oracle,MySQL and other producers)to extract the data to the Kafka message system and the output is processed at the consumer side.