Research And Development Of Cloud Edge Collaborative System For Network Traffic Collection

Nowadays,network attacks become frequent.The attack processes are getting complicated,and the methods are also becoming covert.Traditional network security products such as intrusion detection systems are often difficult to be effective.At the same time,with the vigorous development of big data technologies,artificial intelligence technologies such as machine learning are applied in a variety of cyber security scenarios.It has become a trend to use technologies related to big data to ensure the security of cyberspace.In this process,comprehensive collection of network data is the primary task.With the rapid development of emerging technologies such as cloud computing and the Internet of Things,huge amounts of data in various types generated by millions of network devices and network applications have brought huge difficulties and challenges to data acquisition.Therefore,in this paper,traffic collection,data transmission and processing in the network security big data analysis scenario are mainly focused on.The concepts,development background,technical features,core functions,and architectures related to Intel DPDK,Apache Ni Fi,and edge computing technologies are investigated in this paper.A highperformance traffic collection tool is designed to solve the problems of large data volume and complex data types during the traffic collection process.Combined with the edge computing methods in the field of Io Ts,a distributed cloud-edge collaborative data acquisition solution is proposed and implemented for the problems of stable data transmission,and system node monitoring and management.It has reference significance for integrating resources and improving acquisition efficiency.The high-performance network traffic collection tool designed in this paper is divided into two parts: a capturer and a parser.The capturer is responsible for obtaining the original traffic from the network interface and converting it into specific types of binary data,and the parser is responsible for parsing it into formatted text data.In this tool,traffic capture in conventional and high-speed network environments is supported;the integration of multiple different traffic conversion modules in the form of plug-ins is supported,and three different data output methods are also supported.The data acquisition system based on Apache Ni Fi designed in this paper has a tree structure of 2-3 layers.Based on this,a data acquisition and an edge computing framework are designed.In the three-layer structure,the device layer implements the centralized configuration and invocation of the traffic collection tools,and achieves data acquisition,preprocessing,and labeling;the edge gateway layer implements data routing based on the rule engine and edge computing;The cloud center layer implements different processing of stream and batch data,and at last,the stream and batch data will be written into Kafka and HDFS respectively.The above process completes the data acquisition and transmission,as well as docking with other big data systems in the cloud.In addition,the management functions of the acquisition system in the cloud are also achieved in this paper,that is,the cloud-edge collaboration function of the system.The functions including node status log information collection,system topology structure drawing,and remote control are implemented.The remote control function mainly includes node flow template update,edge computing file deployment,and remote collection control.During the implementation of the above functions,part of the source codes of MiNiFi are analyzed and sorted out.The source codes are modified to make Variables function available in MiNiFi,and meet the needs of functions such as status log collection and topology drawing for the content of the status log generated by MiNiFi.Finally,combined with the system application examples,we show the flow template implementations,usages,and operation effects of the acquisition system functions,such as traffic collection,data transmission,edge computing,data storage,cloud-side collaboration,etc.In addition,the acquisition system was tested from the perspective of function and performance.The results show that the basic functional goals planned in the initial stage of system development have been achieved,but there is still room for optimization and improvement of system performance.