Research On Key Technologies Of Real-time Computing Platform In Cyberspace Security Area

The network becomes extremely huge and complicated in recent years due to the significant increasing of terminal devices,web applications,protocols and network traffic.Malicious network intrusion can hardly be detected and the large scale network attack occours more frequently than usual,people still have a long way to go to make the Internet a safer place.Not only the evolution of the network attack technology but also the computing power becomes the bottleneck for the tradition network security analysis systems to adapt to the complex network environment.The large quantity of input data overwhelms the capaticy of these systems.Firstly,a big data technology based cyberspace security analysis system architecture is proposed in this article,which is made up of seven layers concerning cluster resource management,distributed coordination,data persistence storage and computation model.This paper implements each layer of the architecture with proper open source softwares such as Storm,Spark,Kafka and Flume.This architecture can provide real time computing service,offline batch computing service and graph computing service simultaneously.Based on the architecture,this paper has implemented a real time network traffic monitoring system for Tianjin Education And Research Network(TERNET),which has several data computing logicals deployed on the underlying computing engines.It tracks the real time metrics of TERNET such as flow creation number per minute and packets per flow.Besides,it can detect seven different kind of DDoS attack by performing the detection algorithm against the Netflow data of TERNET.The core of this detection algorithm consists of three different part: Exponentially Weighted Moving Average(EWMA)algorithm,abnormal traffic detection algorithm and DDoS recognition algorithm.EWMA algorithm generates the prediction values of the target metrics based on the history values,which can be used to determain whether the network is in a stable state by abnormal traffic detection algorithm.This paper evaluates the performance of the traffic monitoring system by a set of comparing tests,the key metrics are throughput,Worker's parallelism and Worker's workload.During the evaluation of the DDoS detection algorithm,this paper compares the output of the system and the log file of NSFOCUS Anti-DDoS System,result shows that this system is very accurate in recognising particular types of DDoS attack.