Implementation And Application On Hadoop-Based Network Traffic Processing System

With the development over ten years, theInternet in China has already become one of the most important parts of global Internet. Until the end of June2013, the number of Internet users in China has reached591million, the Internet penetration rate is44.1%. As the rapidly developing of the Internet, the problems are also increasingly exposed. On one hand, the network traffic surges due to the increasing number of users and the endless stream of new applications. And the users claim for higher quality of service in the situation that network congestion is becoming more frequent. On the other hand, because of the complexity of Internet architecture, a lot of problems are lack of deep understanding and accurate description, such as network traffic characteristics, user behavior characteristics and newly-developed application traffic characteristics.lt has seriously affected the further development of the Internet and the efficient use of network resources. At the same time,the rapid growth of network traffic makes traditional traffic analyzing methods face challenge of dealing with huge amount of data. Therefore, new methods which are more efficient and reliable are needed.The Hadoop framework, whose core is MapReduce calculating module, has gradually become a basic distributed massive data processing architecture in cloud computing technology.Firstly, this thesis introduces the basic conception of Hadoop, including the working principleof Hadoop and HBase.Secondly, we propose a fully-functional network traffic processing system with three-tier architecturethat integrates several independent functions, such as the collection, storage, management and analysis of network traffic.Thirdly, we study the data tier of the Hadoop-based network traffic processing system. The Hadoop-based network traffic control system which is the non-real time component of the data tier, and the HBase-based flow log control system which is the real-time component of the data tier, are introduced in detail.Finally, taking the analyzing of smartphone network traffic as an example, we illustrate the application tier.