Design And Implementation Of Data Stream Application System On High-speed Network

Network traffic measurement is to capture packets on the link,analyze and process them to extract the characteristics that reflect the behavior of the network.It is an important prerequisite for analyzing network performance,understanding user behavior and network anomaly and security detection.Analyzing and solving the heavy hitter detection and flow size distribution estimation which are the key issues in network traffic measurement from the aspect of flow length and flow cardinality metrics can support network operation,management and maintenance.However,traffic measurement on the high-speed backbone link needs extremely expensive computing and storage resources,which brings great technical challenge to traffic measurement research.It is necessary to reduce the mass flow data by means of sampling and data stream techniques.The specific work of this thesis contains the following aspects:1.Design and implement a non-cryptographic hash function for data stream methods based on Genetic Programming.As the core of the data stream methods,hash function should meet the requirement of high algorithm performance,hash randomicity and providing multiple independent and irrelevant hash values for the same input.In this paper,we utilize Genetic Programming to automatically design and generate the hash function based on the training sets containing IPv4 and IPv6 traffic.The experiment results show that,GP-Hash can compete with the 17 representative state of the art hash functions on the evaluation indicators we introduced.2.Estimate flow size distributions using Genetic-based EM algorithm based on the statistics of sampled flow.Expectation-maximization algorithm for estimating flow size distributions may converge to a local optimum,be sensitive to its initialization and have high computational complexity.To address the problems,we propose a genetic-based EM algorithm with the advantage of population-based stochastic search ability,which can achieve higher accuracy.3.Design data stream application detection system on high-speed network.It includes two network traffic measurement tasks: heavy hitter detection and estimating the original distribution of flow.To detect the heavy hitters,we utilize Count-Min Sketch with the GP-Hash library to count the packets on high-speed network.From the sampled flow statistics,Genetic-base EM algorithm can be used for estimating the original flow size distribution with higher accuracy.