Http Traffic Analysis Of Web Services Composed By Multiple Resource Providers

Several years ago, service providers provided network services based on HTTP in a centralized way, where a single server bundled with a fixed IP address provides specific web service. Nowadays, architectures of web sites are becoming dynamic and complicated. The very last few years have seen an astonishing development in Content Delivery Networks (CDNs) technology and Cloud Services provisioning platforms. In this complex scenario, contents and services are no longer located in centralized delivery platforms owned by a single organizations, but are distributed and replicated across the Internet and handled by multiple service providers. Understanding the process of complicated web envorionment such as HTTP traffic composition, usage patterns, content location, hosting organizations, and addressing dynamics is highly valuable for network operators. At the same time, the rapid growth of network traffic makes traditional traffic analyzing methods face challenge of dealing with huge amount of data. Therefore, new methods which are more efficient and reliable are needed. The Hadoop framework, whose core is MapReduce programming model, has become the basic distributed parallel data processing technology.Firstly, this thesis proposes a method of classifying HTTP traffic based on correlation coefficient measurement, and presents the mathematical definition and deduction.Then this paper introduces the basic conception of Hadoop, and illustrates a fully-functional Hadoop-based HTTP traffic analyzing system with three-tier architecture, which provides a set of key functions including collection, storage, management and analysis of network traffic.Thirdly, we study the key component of data tier of the system called Hadoop-based IP Address Identification Component, which maps a number of IPs to a service provider. The component is the key component of the proposed analysis system.Finally, we illustrate the distribution of HTTP traffic from a new perspective by processing the classified traffic data.