The Design And Implementation Of Network Traffic Analysis System Based On Hadoop And HBase

With the wide application of network,network architecture becomes increasingly complex and network traffic is continuously growing.In order to manage the network more efficiently,actions need to be taken to analyze the network traffic.Data volume per unit time becomes huge along with the ever-growing network bandwidth.Data size and computation complexity will also become tremendous even for small networks as sampling time increases.It becomes more and more difficult to make rapid analysis with traditional network traffic analysis technology.It is time to introduce more efficient and reliable big data oriented processing methods to perform analysis on network traffic data so as to detect abnormal network traffic such as DDOS attacks,Worm Virus etc.and finally to offer a powerful tool for network management.In this context,the thesis designs a big data oriented network traffic analysis system,which takes Hadoop HDFS(Hadoop Distributed File System)as underlying storage service,stores network traffic data in a distributed manner with HBase and provides interactive query features.It shows the characteristics and changing trend of network traffic data visually with figures and realizes the storage,retrieval and analysis of network traffic data through connecting HBase and R language with rhbase.Specific work done in this thesis is as follows:(1)System architecture design.By screening values of Hadoop ecological system components,adopting Hadoop and HBase,introducing visualized analysis of big data and exploring big data applications,this paper designs and implements a distributed network traffic analysis system according to the network traffic data characteristics and requirements.(2)System platform construction.A system running platform is set up by building a Hadoop and HBase cluster,which uses Zookeeper to maintain its stable operation,on Linux.(3)System features implementation.Network packets are captured with Libpcap and stored into HBase with Thrift.HBase table structure and design of RowKey are optimized in order to realize highly efficient storage and query.(4)Visualized analysis.Abstract network data are shown graphically with three-dimensional scatter plot,which is selected based on information entropy,according to the characteristics of network traffic data.And then,it obtains the overall network status and completes analysis on network traffic data quickly using figural perception of human vision.(5)System testing.Tests are performed on key targets such as throughput,delay and so on.Actual system performance is examined and system design is validated.Design of the system is finally proved reasonable and feasible.The system meets practical requirements.