Research And Application Of Kafka-based High Speed Traffic Storage And Distribution System

With the national Internet+ strategy proposed,the impact of the rise and fall of the Internet industry is not only confined to the IT industry,but also gradually become a major event related to the national economy and the people's livelihood.Considering the Internet as a driver to stimulate industrial innovation and promote cross-border integration,thus driving the entire economy has become an important direction of our future development.At present,China's Internet industry is gratifying,the number of China's Internet users has reached 649 million at the end of 2014.However,with the rapid development of the Internet industry,the problem is also increasing.On the one hand,as a variety of new business increase,as well as the size of the user expand continuously,the traffic of the Internet has increased,thus it's a huge challenge to ensure the quality of services.On the other hand,as the Internet has more intimate contact with the user,more and more user data is acquired by us,but how to extract the real value from such a large number of data has become more and more difficult.Facing these problems,the traditional data processing tools are far from meeting our needs.In order to cope with this situation,our target is to introduce a data storage and distribution system which can deal with the high speed traffic and propose a more perfect processing mechanism.Firstly,this thesis introduces the basic functions of Hadoop and the working principle of Kafka,especially its important role in complex system.Secondly,on the basis of Hadoop technology and Kafka component integration capability,we propose four layer system structure of network traffic processing system,which integrates network traffic storage,distribution,processing and analysis.Thirdly,a detailed performance test in the Kafka component has been done,which is in the core position of data distribution in this framework,in order to guarantee its application performance in large traffic and high speed scenario.Fourthly,this thesis focuses on the data layer of the network traffic processing system.The off-line component of the data layer:Hadoop based network traffic data control module,and the real-time module:Storm based stream record control module,are introduced in detail.Through the research of these two components,some important problems of mass network traffic analysis are solved.Finally,we take the DNS analysis system and user community analysis system as an example to demonstrate the good performance of the system in network traffic monitoring and user behavior analysis.