Research And Implementation Of Real Time Traffic Analysis Technology Of Large Scale Networks

As the fast expansion of Internet and the exponential increment of network bandwidth, real time traffic analysis of large scale network faces the challenge proposed by massive traffic data arriving fast and randomly.Streaming data mining, as a new research area of very large data processing, gives a new way to analyze network traffic data. Different with traditional data mining algorithm, streaming data mining algorithm scans the data only one time, gives data synopsis quickly, and provides continuous query results. Probably applied in network traffic analysis, it can quickly analyze traffic data, giving result of predefined query, which can greatly benefit network management, intrusion detection and fault analysis.Based on the characteristic of Internet traffic and multistage filter, this thesis gives a new algorithm detecting large flow by separating large flow traffic. The algorithm effectively reduced false positive errors and memory accesses. Theoretical analysis and experiments showed that it can quickly detect large flows using only few memories and memory accesses. In addition, this algorithm can be used in large networks. The memory requirement could be predicted. The precision of the algorithm could also be well controlled when applied to large network.After building a traditional network management system—NetWatcher, we successfully applied streaming data mining algorithm to real time traffic analysis which gives on time traffic data summary while avoiding saving massive data into database to reduce storage cost. The system also provided an open platform for further integrating new stream mining algorithms.Finally, NetWatcher was used to manage the interconnection network of massive storage system and played a very important role. The system could find the large flow in the traffic as well as giving SNMP, sFlow traffic information.