Realization And Application Of HTTP Data Acquisition System

Nowadays computers and network popularize and penetrate thousands of families. We have ushered era of DT(Data Technology). As most of the network packets are dominated by the Hypertext Transfer Protocol, clollecting packets in real-time and efficiently and digging for information can help us perceive data value in real network and know behavior of Internet users. Thus we can optimize network and provide targeted network service to people.In this paper we introduce how HTTP works in World Wide Web,packets features and distributed data processing platform firstly which provide theoretical basis for subsequent system development and data analysis. Secondly we design and implement HTTP data acquisition system which is able to collect packets from netwok in real-time,restructure packets out of order, parse packets according to needs of users using multimode matching algorithm and generate logs. The system has passed function test and performance test. In order to fit real network better,we analyse factors that affect performance and optimize the system on structure of software and parsing packets.We deploy the system in core network and campus network to collect data in real-time. We analyse data in both network and understand characteristics of different users and packets features of HTTP. Besides, we analyse user behavior in Douyu application. We apply data analysis to commercial software to help understand users. Last the paper gives a summary and outlook on HTTP data acquisition system.