Research And Implementation On Traffic Classification And Mass Flow Logs Analysis System

In recent years, the Internet in China, especially the mobile Internet, was rapidly developed. Until the end of June2012, the number of Internet users in China has reached538million, the Internet penetration rate is39.9%. Network traffic monitoring has become an important technical measure to ISPs for network management and operation. With the diversification of network applications, the identification and classification of the network traffic is facing grand challenges. Research on identification and classification methods which can achieve high accuracy and low error rate has become a hot point. With the increasing of the network speed, the size of network traffic data increases sharply, the common analysis method has been unable to meet the massive traffic data analysis needs. Google’s MapReduce programming model has become an important method for massive data analysis, and then Hadoop cloned this model and has been recognized by both academia and industry. Hadoop has become an important tool of massive data analysis.This thesis first introduces the network traffic identification and classification techniques, including deep packet inspection and the deep flow inspection methods. Then, the massive data analysis platform, especially Hadoop system and its application in flow analysis are introduced.We have developed a network traffic analysis and classification system (TACS) based on the research on traffic identification technology. This thesis describes the main function of TACS, the overall design of the program and the key sub-module design description. In order to analyze the vast amounts of traffic data, we developed a hadoop based system, LogAnalyser, making the processing and analysis of massive data become quick and easy. This thesis describes the main function, the design scheme of overall and key sub-modules of LogAnalyser.Finally, the ADSL and CDMA network traffic characteristics of P2P streaming application, GPRS network services distribution and network quality characteristics are analyzed using TACS and LogAnalyser.