Real-time Data Synchronization Mechanism Based On Buffer Optimization

Under the background of cloud computing and large data, the data presents an explosivegrowth trend. The information system based on traditional architecture, has been difficultto deal with the vast amounts of unstructured data, such as video, images and text. On theother hand, traditional business intelligence and data analysis system are also lack ofeffective analysis tools and methods. Considering in business value and technicaldifficulties，the keys of big data are real-time processing and data mining. So how to fetch,process and synchronize real-time data are hotspots in current research.According to the function and performance requirements of the real-time datasynchronization system, we propose a real-time data synchronization mechanism based onbuffer optimization. The system includes four modules: data capture module, data parsemodule, data processing module and data service module. Based on the analysis in the loglogical replication technology, data capture module captures the log files of heterogeneousdatabases, and parses those logs into the binary data stream. During the period of dataprocessing, data parse module uses Google Protocol for message encapsulation. Finally,data service module maintains multichannel distributed buffer coherency by leveraging thesynchronization techniques in Hadoop Zookeeper. The system can achieve thesynchronization among heterogeneous data sources and support the multiplex dataconsumption.With buffer optimization strategy, the system not only enhance the stability, reliabilityand high availability of the databases, but it also reduces the delay time of accessingreal-time data in peak loads. In the overall testing, compared with the traditional way ofdata persistence, the results show that the buffer optimization techniques improves thesystem reliability, stability and database access performance during periods of peak loads.Moreover, the buffer significantly reduces the delay of real-time data access by users,achieves the synchronous speed in the0.1second period of time.