Research On Network Telemetry Data Flow Analysis And Optimization Methods

With the advent of the big data era,data centers have become an important infrastructure.Network security threats emerge endlessly,and data center security has become the focus of everyone’s attention.Real-time security monitoring and analysis for data centers is one of the important means to ensure network security.However,the east-west traffic of the data center reaches 100 G or even 200 G,which puts forward higher requirements for real-time network flow data analysis technology.In this regard,the industry has introduced network telemetry technology for network status monitoring.The network telemetry data contains comprehensive and real-time network information,which lays the foundation for large-scale network real-time abnormal monitoring and online threat detection.This article focuses on the problems of largescale network telemetry data flow analysis,such as high data analysis overhead and single detection task.The main work of this paper is as follows:In view of the large-scale telemetry data stream analysis,on the one hand,the concurrency and scalability of the detection task in the network telemetry data stream analysis is insufficient;on the other hand,with the increase of network detection tasks,the pressure on the query analysis processor also increases Question,this paper proposes a network telemetry data stream classification prediction method for network threat detection.According to the behavioral attributes of different network threat detection tasks,this method constructs a set of lightweight classification prediction models,so that in the subsequent threat detection process,only a subset of data related to the threat needs to be paid attention to,thereby improving analysis efficiency,and finally distributed in Spark Streaming Parallel analysis of detection tasks is realized on the stream processing platform.Experiments on the CIC-IDS2017 and WIDE public data sets show that when the accuracy rate reaches 95%,compared with the network threat detection in the full data,the method in this paper reduces the total processing time by an average of 15%,and with Spark With the increase of computing nodes,the classification prediction method has good scalability.Aiming at the problem that the distributed stream processing platform Spark Streaming performs network telemetry threat detection tasks without setting reasonable parameter configuration during large-scale telemetry data stream analysis,its performance is not optimal,and a method of Spark Streaming parameter optimization based on deep reinforcement learning is proposed.,Which proposes a weighted state space transfer method,which allows guided parameter space transfer when exploring the environment,and speeds up the training of the model.Optimize the parameter configuration by studying the relationship between the configurable parameters and the running performance of the stream job.Experiments show that the total scheduling time is reduced by 99.72% and the total processing time is reduced by 57.39% when analyzing the network telemetry data stream with the default parameter configuration.It also has a good effect on two general flow tasks.