Research And Implementation Of Network Security Log Analysis System Based On Spark

Different kinds of Network Security Appliances(NSA)are installed in the gateway to ensure the safety of the enterprise's intranet.These security devices,such as firewalls and Instruction Detection Systems(IDS),will generate large volume of logs to record network events.The logs of the security devices have the following characteristics: multi-source heterogeneity,magnanimity,complexity,and spatial-temporal correlation.Traditional network security logs analysis technology is increasingly difficult to meet the needs of logs,which in collection,storage,pre-processing and analysis.Big data has 4V features: Volume(large),Velocity(high speed),Variety(diversity),Value(value).The purpose of this paper is to develop a fast,good scalability,extensibility,a large quantity of processing data network security log analysis system,from massive log collection,storage,preprocessing and analysis of four aspects to introduce the Spark for the core of big data technology based on the characteristics of the network security logs,so as to achieve large capacity,low cost,high efficiency network security analysis capabilities.Firstly,this paper describes the challenges and difficulties brought by the massive security log analysis to the network management personnel,and then introduces the advantages of introducing big data technology from the aspects of log collection,storage and analysis.We complete distributed acquisition and transmission of massive network security logs by using Flume and Kafka technology,and carry out the function secondary development of Flume source code according to the actual demand.Secondly,we design and develop network security log preprocessing module based on Spark SQL component,which called ETL_ON_Spark,and based on it,we successfully completed the preprocessing of network security log : filtering,normalization,aggregation.At the same time,On the basis of using Spark SQL and Spark Streaming technology,and combined with monophyletic log feature matching and multi-source log correlation analysis method,we complete the analysis of network security logs,including abnormal traffic real-time alarm and network attack events alarm.And then we develop a web display and query interface to meet the needs of network management personnel on visual display and query of network abnormal events.Finally,we carry out functional tests and performance tests of the key modules of the system,it is proved that the network security log analysis system based on Spark has initially met the design goals and requirements of this paper.The main innovations of this paper are as follows:(1)In this paper,we propose a clustering algorithm based on the attribute dissimilarity of network security logs.The proposed algorithm is on the basis of the features of logs that corresponds to the network attack model,and it assigns different weights to the importance of different network attacks,according to IP address,port numbers etc,so as to complete network attack alarm repeat log aggregation.Experiment has shown that this method has a good aggregation effect for the repeating alarm log.(2)By using Spark SQL rich external data source interface and its abstract programming interface DataFrame,we propose and develop network security log preprocessing module: ETL_ON_Spark.Compare to Traditional log preprocessing technology,The ETL_ON_Spark implemented in this paper has the advantages of processing mass data,quick speed,good scalability.