Network Intrusion Detection Based On Flow Of Distributed Database System

With the high-speed development of Internet, security is an important topic.The traditional network security for users and enterprise users, the main use of technology including the system intrusion detection, antivirus software and firewall. This kind of security measures usually does not reduce mass communication network(i.e., the backbone network) of abnormal traffic. In order to reduce the abnormal flow in the network, to reduce or eliminate various attacks suffered by the user, mass communication network routing exchange equipments must have abnormal traffic detection and recognition ability. For flow of abnormal operation has the following several ways: a) determine whether there is any abnormal traffic, this call flow detection.B) the type of traffic anomaly, which is called traffic identification. Current flow testing in accordance with the granularity is mainly divided into three types, respectively is: based on the package, based on the flow, based on the traffic.This paper proposes a more granular, based on dynamic session window to aggregate the IP Flow filtering algorithm and combining the characteristic of the netflow records using the One-class SVM algorithm to detect DoS attack. At the same time in order to support screening characteristics calculation operation, this paper extends the Spark Stream to support of SQL query operations on the Stream. In the process of thesis research, the existing feature selection algorithm, the Spark Stream and working principle of the Hive, and the selection of kernel function of SVM has carried on the investigation and research, for the current flow testing carried on the thorough understanding. First of all, the traditional feature selection algorithm based on entropy, is to put the different IP source together to calculate the entropy of information, there were some defects, when an exception occurs, it is hard to know who the aggressor, who the target is. Paper based on sessionkey(srcIP,desIP,Portsrc,desPort)to aggregate different flow data records, and then obtain network flow data of information entropy as the training feature to solve the problem. The present study showed that abnormal flow accounts for the total flow rate and the detection effect is positive correlation, when the abnormal flow of low detection effect is very poor, so this paper also tries to find the solution of the problem, this paper puts forward the session window way solve the problem. In the face of sudden large data sets, the lack of support for the ground floor of the main calculation model, and is not efficient with anomaly detection algorithm, providing SQL operations on the Stream, and support the continuous queries and window operations by extending Spark Stream.Finally, this paper compared the feature selection algorithm proposed in this paper to traditional ID3 and C4.5 algorithm, good or bad for feature selection results of judgment, the most direct and effective evaluation criteria is the algorithm selected feature subsets and the optimal feature subset of similarity. But in practice, the optimal feature subset without evaluation criteria, therefore, in order to verify the effectiveness of the proposed feature selection algorithm, in this paper, an indirect verification method to be used by the selected feature subsets in One-class SVM classification algorithm in the AUC indicators to measure the stand or fall of feature selection. This paper simulates the abnormal flow of different proportion of total flow by the window, to show that based on the session window feature selection algorithm is stable under different proportion of abnormal flow. At the same time, the result of experiment shows that this article is based on the Spark Stream of SQL extensions, can complete computing needs.