Research On Key Technology Of Hadoop-based Network Security Log Audit System

Network traffic log contains everything that occured in the network.Analysising ofnetwork traffic logs can not only locate security events in the network environment, but alsofind the abnormal behavior of internal users in the in network. Most of the traffic anomalydetection systems are real-time and using static or dynamic threshold to monitor the entirenetwork, and can only be delineated certain time window, not can not identify the cyclicalexception. Offline audit log system is nessesary to network security event audit. With theincreasement of network applications, network tranffic log generated rapidly, the distributedprocessing has been widely used up. Hadoop is an open source distributed systems, which canbe used to deal with the massive network tranfic log. How to take advantage of Hadoop tomake reasonable data storage and how to combine the traditional audit approach withMapReduce to complete the audit task are two problems to be solved.In this paper, Hadoop related technologies as well as traditional outlier detectiontechnology are the basics. We start work in the following areas:First, this paper presents an alaorithm of Hadoop pretreatment data scheduling algorithmbased on IO load. In Order to get organizational data, the data is stored in Hadoop HBase. Weuse MapReduce to preprocess data. The Hadoop HBase original data scheduling algorithm isbased on the balance of the number of data blocks, this algorithm can only be guaranteed foreach Server has substantially the same number of data blocks, and does not consider theutilization of the data block. Consider the IO load of the data block to the data scheduling canbalance the resource utilization of the entire cluster, to alleviate the excessive utilization ofcertain Server. At last, the experiment veries the correctness and validity of the algorithm.Secondly, this paper presents an audit algorithm which combines outlier detection andMapReduce. MapReduce and Outlier detection technology are used to analyse networktranffic log. Audit of the same time period for each user in different days, to make up forreal-time network traffic anomaly audit audit only for the whole network traffic, not for eachuser based on the deviation of their own habits.At last, the experiment veries the availabilityand correctness of the algorithm.Finally, we developed an audit system based the Hadoop to store and analysis the network traffic log and enevtrally find the abnormal flow. We describe the whole MapReduceprocessure of the system. We illustrate overall framework and core modules of the system.