Design And Implementation Of STORM-based Real-time Network Analysis System

In recent years,with the accelerated pace of mobile Internet infrastructure and rapid development of network traffic,especially mobile network traffic,the traditional data processing tools have no longer been able to meet the processing requirements of large amounts of data.Massive data brought great difficulties and challenges of collection,transmission,storage and computing of data.Hadoop,a software which Google developed,was popularized rapidly.With Big Data ecosystem and Real-time computing of big data was increasingly demanded,large numbers of tools based on Hadoop such as Pig,Hive flourished.Company Twitter officially released a real-rime data processing tools in 2012 named Storm,which solved the problem of real-time processing of big data.Apache Storm is a free and open source distributed realtime computation system whose code is hosted on GitHub.Storm developed rapidly with urgent needs and has been applied to many other fields.In China a lot of companies have their own Storm clusters.Storm has become an indispensable tool for distributed real-time computing and an important means of real-time processing and analysis of massive data.This thesis first introduces the quality analysis of network and current research status,then introduces the significance and difficulties of network quality analysis.After that we highlight the architecrue of Storm and the key technologies it uses including the function of its primary components and its reliability.Then this thesis designs and completes a Storm-based network traffic real-time analysis system.Finally we analyze the performance of the system and propose some improvement schemes.In consideration of the feature of network data and the demand of network quality analysis,this thesis put forward a design which use Kafka as middleware,use Storm to process data and use PostgreSQL to store and display data.We use Storm Pluggable Scheduler to achieve point-to-point transmission between Kafka and Storm,which uses network IO maximizely.Then we use a variety of scheme to improve the efficiency of data processing and the throughput.The whole system implements the collection,processing and storage of big data and provides a reliable implementation of network traffic in real-time analysis.