Research And Implementation On APT Modeling And Netflow Pre-processing Based On Hadoop

Advanced persistent threat(APT)is a form of intrusion,which some intruders who is proficient in complex technologies use a variety of intrusion vectors(such as network,physical and fraud)and rich resources to create the opportunity to realize the purpose of their own.In recent years,APT frequently occurred,such as RSA Secur ID,Night Dragon,etc.,which pose a great threat to the enterprise and social.So Research on APT detection technology is imperative and research on APT detection technology is a key research point in the field of network security.This paper modeling analysis typical case of APT invasion-“Night Dragon” based on UML modeling language,which includes Night Dragon's case diagrams,sequence diagrams and activity diagrams,etc.On this basis,paper analyzes the general rules and characteristics of APT intrusion.And then distinguishes it from traditional intrusions.The findings section show APT intrusions generally lasts long term and has better concealment.So the requirements of APT detection are higher,which includes the time span and space extent of detection-data source,complexity of the detection algorithm.By modeling,we found that the detection of APT need to analysis large amounts of network traffic data.Because the amount of data in the original network traffic is too large,its analysis needs a large amount of system resources,and the analysis efficiency is relatively low.Therefore,in order to detect well for APT,in this paper the original data of the Net Flow stream will be pre-processed for APT analysis and detection.All network traffic of one internal network is pre-processed.The pretreatment of Net Flow is studied in this paper,which mainly includes the collection and aggregation of all the Net Flow stream data in the network.In this paper,we design a collector which can collect the Net Flow stream efficiently,and reduce the packet loss rate based on the multi-link queue.At the same time,because the Net Flow stream is faster and more quantity in the busy network,the analysis can be carried out according to some rules to reduce the amount of data analysis and improve the analysis efficiency.On the base of Net Flow stream,multi data granularity based on time is designed to meet the needs of analysis.Then according to the characteristics of network traffic,several Net Flow stream aggregation strategies are designed,which includes Net Flow stream aggregation based on IP,Net Flow stream aggregation port and Net Flow stream aggregation based on protocol.Hadoop is the mainstream of the big data analysis platform currently,it is a distributed processing architecture.In this paper,we design the system structure of Net Flow stream,‘Map()' function and ‘Reduce()' function based on Hadoop data analysis and Map-Reduce distributed computing framework.