Real-time DDoS Detection System Based On Spark Streaming

The birth of the Internet brings progress and convenience to all aspects of society,but also brings a variety of problems.Network security is one of them.Distributed denial of service attack(DDoS),as a traditional network attack mode,has always threatening the security of the Internet.With the rapid development of the Internet,it is changing constantly,whether in scale or in attack form.From the manual deployment attacks in the 1990s to the T-level attacks,DDoS has reached an unprecedented level of complexity and quantity.Moreover,it has been favored by attackers because of its low cost,good effect and far-reaching impact.In 2013,the peak attack traffic of DDoS was still 300 Gbps,and by the first half of 2018,it had reached 1.7 Tbps.Although the broadband consumed by DDoS is only part of its aggression,its rapid rise reflects the growing threat of DDoS in the context of data streams processed by most companies'Internet infrastructure below 100 Gbps.In addition,with the gradual entry of traditional industries such as medical profession and education into the Internet,these fields have also been attacked to varying degrees and the number is on the rise.Therefore,with the constant changes of DDoS attacks,new requirements are constantly put forward for its attack detection and defense technology.The traditional DDoS attack detection is difficult to meet the needs of processing efficiency when facing the attacks of GB and TB magnitude,and the lack of real-time performance of traditional methods is also a major deficiency.In practical applications,the lack of real-time will seriously reduce the user experience of normal users,and the damage is often unbearable.In order to solve the shortcomings of traditional DDoS attack detection system in the face of large traffic attacks and real-time,this paper designs and implements a real-time DDoS attack detection system based on Spark streaming and Apache Kafka.The system acquires network flow in real time and pushes it to Kafka after preliminary processing to facilitate subsequent high throughput processing.Spark streaming is used to subscribe data packets from kafka,filter them,and then process them.Then,Entropy-based DoS detection algorithm is used to calculate and detect them.Finally,the test results are presented to the client in real-time visualization.The system test shows that the system can detect DDoS efficiently and effectively deal with large-scale traffic attacks.