Research And Implementation Of Big Data Transmission Technology Based On WAN

With the development of network technology, as well as the improvement ofcomputer processing ability, Internet products spring up quickly, and the world isgradually towards a state that all things connected. However, the consequent mass databring many troubles to researchers. These data mean not only large capacity, fastgeneration, and great varieties, but also distributing in various regions, so people oftenrefer these data as big data. A large number of applications derived by big data, such aspublic opinion analysis, weather forecast, and advertising, which have importantcommercial value and research value. The efficient transmission of big data in the WAN isnot only the premise of the big data applications, but also the basis of big data processingand analysis. The traditional data transmission methods may be more mature than thenews, but in the age of big data, these methods have many limitations, such as tedioustransmission process, low transmission efficiency and no adaptation to the multi-sourcedata transmission.This paper does research deeply on the technologies related to data transmission, andmainly resolve the problems of instability and low efficiency in the transmission. Thispaper selects TCP as transfer protocol, and designs an efficient packet in the applicationlayer which resolves the problem of complex transmission method. Based on the analysison the research result of data transmission and WAN Acceleration, using the method ofpush and pop, this paper designs a message queen driven by event to ensure the reliabilityof data transmission. According to the characteristics of the WAN and big data, using theidea of automatic adjustment, this paper puts forward an effective method to do resourcecontrol and cache control which improves the efficiency of data transmission. Accordingto the research result above, this paper completes the design of communicationmanagement module, datagram encapsulation module, node data sending module andcenter synchronization module by using JAVA and C programmer, then build a big datatransmission system based on message queue. Through function and performance test tothe transmission system, it proves that has certain advantages in transmission rates andnetwork packet loss rate compared with other transmission system.