Design And Implementation Of Log Real-time Monitoring System Based On SparkStreaming

In recent years,with the continuous development of Internet and cloud computing,more and more people have begun to contact the Internet.IT companies business also began to rapid growth,they produce a lot of log data every day,and the existing enterprise logs are generally stored in a database or file,and then IT offline processing,this way is not suitable for enterprise business according to its characteristic calculation of the real-time monitoring and always need to get the next day as a result,the calculation speed is very slow.Therefore,in addition to the characteristics of some existing log computing systems,we will provide real-time and comprehensive monitoring of data on its basis.This paper studies the principle and Map/Reduce SparkStreaming real-time streaming data processing programming model,and then combined with the analysis of the existing common logging system,puts forward the SparkStreaming real-time monitoring system of the functional requirements and non-functional requirements.Next,the overall architecture of the monitoring system are introduced and the realization of the function of the module,finally through the Docker virtualization implementation enterprise level of large-scale cluster deployment and test to the system in detail,and illustrates the result of the test and analysis.The system not only has realized the log collection,log processing,log storage,and other functions,but also support a variety of data sources a variety of different structure of log,alarm rule configuration,log alarms,and other functions.The system also provides an open interface that allows developers to bury applications in the system to perform large-scale log real-time computing and storage tasks.The system can expand the computing cluster horizontally through the management of the Spark cluster,thus avoiding the common single point failure problem of the previous log system.