Design And Implementation Of Real-time Log Stream Processing System Based On Kafka And Storm

Real-time computing of massive data is an important aspect of the big data technology.Real-time log data analysis has become increasingly popular demand for large Internet companies.The real-time processing technology of log message streams have a wide range of applications,for example,personalized recommendation,real-time risk control,users' prediction of interests.The huge amount of log data contains significant value.And Its utility is becoming ever more timely.Traditional log off-line processing is no longer applicable to the specific data scenarios and requirements.Instead,Real-time streaming data processing technology is designed to service latency-sensitive business systems which require real-time data to be computed and analyzed.This system aims to design and implement a real-time log stream processing analysis system based on Kafka and Storm to achieve real-time and efficient processing of log data.The system combines the characteristics of the log stream.Through the integration of Flume,Kafka,Storm,Tair and many other big data technology,the system designed and implemented a completely distributed,stable,easily-extensible,high-performance Real-time log stream processing system.This system solves a set number of technical difficulties,such as heterogeneous log data collection,real-time log data analysis,Storm intermediate computing results storage,Storm and Kafka integration development.And this service optimizes the source code for Kafka and Storm technology to address system performance issues after dual-cluster hybrid deployment.The system development and implementation of relying on a domestic groupon business website.The primary responsibility for the system is to analyze the ad log data in real time and to provide data support for systems that depend on advertisement log data.