Research And Implementation Of Key Technologies Internet Massive Traffic Information Collection And Transmission

With the rapid development of China's Internet,the link between the national economy and Internet technology innovation has become increasingly close.The number of Internet service customers is increasing every year,and the number of terminal devices connected to the Internet is also increasing.More and more enterprises are relying on the Internet to expand their business.Applications derived from massive data generated by various types of data terminals,such as customer behavior tracking,public opinion analysis,and advertising,have high commercial value and scientific research value.However,due to their large capacity,rapid speed,wide variety and cross-regional characteristics,these data are extremely demanding on the level of acquisition and transmission technology,which has become a difficult problem for researchers.Big data collection and efficient transmission in the WAN(Wide Area Network)are the basis of big data applications and the basis of big data analysis and processing.The system needs to be designed according to the application scenario.Firstly,this thesis introduces the background knowledge of message queuing technology and load balancing technology in big data transmission,and elaborates the architecture of data acquisition and transmission system.It also carries on related technologies involved in the system,such as distributed coordination technology and storage technology.The message queue optimization scheme is proposed in this context in detail.Secondly,the key technology of load balancing in file server cluster is explored.After analyzing the advantages and disadvantages of traditional load balancing algorithm,a new dynamic load balancing algorithm is designed.It solves the problem that traditional algorithms do not perform well in heterogeneous clusters and have insufficient sensitivity to real-time load changes.Finally,the scheme proposed in this paper is compared with the corresponding technology in traditional data transmission scheme in terms of load balancing effect and the real-time effect of data.