A Malicious Flow Monitoring And Analysis System Based On Spark Platform

In research on DDos monitoring, various methods are widely mentioned, but they are all targeted at certain specific areas of the scene., lack of universality; and in the field of big data platform, appears some good platforms, such as Hadoop, Spark, etc., but they do not directly support malicious traffic monitoring; the existing tools and existing research of Netflow have some good tools of malicious traffic monitoring, but do not support monitoring of DDos. If we can combine the advantages of the technique research with that of the technology platforms, the defense against DDos may be more effective.The main results of this paper are as follows:Firstly, we propose a malicious network traffic monitoring system model based on Spark platform,focusing on monitoring DDos attack, we develop relevant principles. Feature construction method is proposed based on linear fitting Based on this, we improve four monitoring algorithms based on machine learning algorithms. Secondly, we build a malicious network traffic monitoring platform based on Spark, including Hadoop platform and Spark platform. Thirdly, the Spark platform and Hadoop platform, respectively, achieved the above-mentioned four improved algorithms, and experimental comparison. Select the optimal algorithm and explained it.Machine learning algorithms in feature selection, Often only considered independently associated attributes, this does not reflect some of the correlation. We linear fit the number of traffic flow and service and select residuals as a feature, taking into account the comprehensiveness, the average number of packets and average packet size is also a feature, so you can simulate normal traffic characteristics to a greater extent. Based on new features, we improve four algorithms based on machine learning, They are kmeans, decision trees, Bayesian learning and PAM. Using the Spark machine learning algorithm interface, we developed corresponding algorithms, in determining the center of the class,kmeans tryed a different method.This paper analyzes the feasibility of machine learning algorithms on detecting the malicious traffic, analyzes the applicability of Spark platform for machine learning algorithms, combines with malicious traffic detection ways existed and algorithms above, a malicious traffic detection platform is created that can detect worm,Trojans, botnets and DDos attacks more comprehensive.We create a new feature extraction method, improve the machine learning algorithm When users want to use, they can select different modes according to the requirement of time bound. On the basis of the extraction method based on the linear fit, the improved algorithm has been tested, compared and analyzed.