The Research On Data Storage And Traffic Modeling In IP Network Measurement

IP networks, INTERNET as one of their representations, are becoming basic facilities of our society. However, the IP networks' properties are not entirely completely known because of their large scale and complexity. Traffic measurement and modeling is essential to planning and management of existing IP networks, as well as to designing next generation networks. However, the problem of measuring and modeling traffic of IP network is difficult due to the nature of traffic. Complexity and diversity of traffic are constantly growing, and the increasing network traffic is always dynamic. While there is a lot of research on characteristics of IP network traffic in the world, but only a little in China.Based on the Traffic measurement system developed by ourselves and several network tests and measurement projects, this paper collects data from the traffic of IP networks in China and is concentrated mainly on such questions as the storage and share of traffic measurement data, traffic modeling and forecast and traffic anomalies detection. Also, this thesis contains the background material and literature survey, where it provides an overview of pervious work and findings on the measurement and modeling of IP network traffic. The main contributions of the thesis are mainly comprised of four parts:The design of traffic measurement system and OPC interface.NetTurbo, a high performance traffic measurement system, supporting traffic monitoring from data link layer to application layer, is developed by ICT and HunNan University cooperatively. The author participated in the development and took charge of the design of modules including passive measurement, realtime and historical data analysis. In this paper, in addition to existing technology such as MIB, OPC interface is firstly employed to make the traffic measurement system more open and convenient to transfer data to other OPC client tools for further analysis. Through OPC interface, other network service systems are able to get the data when they need to do service control from the measurement platform easily.The design of multi-tier flow storage system.Real network traffic data is important for our research. Several network measurement projects were carried out separately in the China Unicom,GreatWall Broadband Network,China Construction Bank, China Telecom from year 2005 to 2007. A great deal of useful data was collected in these projects. The paper describes the design and implementation of a multi-tier storage system to efficiently support network traffic data recording and retrieval. An efficient traffic data storage approach different from the method used in Cisco Netflow is provided by leveraging the heavy-tailed nature of network traffic:because the bulk of the traffic in high-volume streams comes from just a few connections, by constructing a filter that records only the first N bytes of each connection we can greatly winnow down the recorded volume while still retaining both small connections in full, and the beginnings of large connections. To improve the application performance, the base string functions are rewritten in assembler language according to the newest CPU microarchitecture.Traffic prediction models of traffic at application layer in metro area networksThe current network traffic measurement research is mainly concentrated on the flow forecast and analysis based on those at network layer or transport layer, but a single ARMA (n, n-1) model which can only describe the overall network traffic trends is used, and different traffic at application layer aren't always consistent with ARMA (n, n-1) model. This paper presents traffic prediction models at application layer, which use ARIMA seasonal multiple model (p, d, q) (P, D, Q) s for modeling and forecasting the seasonal time series from China's exports of a metro area network link. Experimental results show that different application layer traffic presents different traffic behavior characteristics, forecasting trends are consistent with the actual flow curves, and Mean Absolute Percentage Errors are around 10%. To the best of our knowledge, it is the first time to present ARIMA seasonal multiple model traffic prediction models at application layer.Traffic anomalies detection based on multivariable statistics analysisAnomaly detection and analysis based on network traffic are used to monitor and locate network anomalies for network and security management. Recently network anomaly detection using dimensionality reduction techniques has received much attention in the literature. Network-wide anomaly detection based on Principal Component Analysis (PCA) and Origin-Destination flows (OD flows) has emerged as a powerful method for detecting a wide variety of anomalies. In this work we develop an approach for anomaly detection for large scale networks such as that of an enterprise or an ISP. Our traffic matrix is not built with the OD flows but the traffic metrics provided by the network measurement system developed by ourselves, and PCA is also applied to reduce the dimensionality of this matrix. The method divides the traffic metrics matrix into normal subspace and anomalous subspace. SPE statistic is used to detect anomalies, and the anomalies are analyzed with contribution plot of SPE statistic. Experiments show that traffic anomalies can be effectively monitored with SPE, and the main causes of the anomalies can be found out with contribution plot.