Research And Implementation Of The Distributed Traffic Performance Collection And Mass Data Analysis System

Recent years, as the flourishing growth of Internet technology, more and more Internet applications came into being. The complication of network topology and network environment and the aggravation of network workload lead to problems of network security and the quality of service, such as the decline of network performance. To solve these problems and plan for new network traffic, it is necessary to analyse network performance indicators and monitor and assess the quality of service of the network, so as to evaluate the overall performance of the network.Research of network measurement emphasizes on how to do statistical analysis on mass measurement data, from which the overall network performance can be derived. In current techniques, most of the operations of data analysis of network measurement use a centralized management platform. The integrity and reliability is not guaranteed by an isolated management platform collecting and analysing the mass measurement data from multiple measurement probes. On the other hand, when facing mass measurement data, the centralized platform is not capable for analysing mass data due to its limited computing ability. Nowadays, as the spread of cloud computing technology, using the powerful computing ability of cloud computing platform to process mass data has already become the trend of research and development in network measurement.The Distributed Traffic Performance Collection and Mass Data Analysis System resolves the problem of rapid calculation of mass data by the MapReduce computation model of Hadoop platform, and utilizes performance evaluation algorithm to analyse the overall performance of network. Mechanisms of task assignment and fault-tolerance for node failure are added based on the principle of task assignment and fault-tolerance of Hadoop framework.In this paper, first the statuses of relevant research in the field of cloud computing and network measurement are introduced, together with network performance metrix and network performance evaluation methods in a nutshell; second, requirements analysis is done according to problems encounterred in real network measurement, while functions are divided and performance demand is completed based on certain design principles; third, design and implementation are done in the light of requirements analysis from general framework deep into detailed modules; last but not least, testing environment is set up to test each funtion realized in this paper, and to prove the capability of this network measurement system in a distributed network environment by analyzing the results of tests.