Design And Implementation Of High Reliability Data Incremental Synchronization System

With the improvement of social informatization,enterprises and social organizations generate more and more data in daily production and operation,and at the same time,they are paying more and more attention to the management and protection of data.In this context,the market demand for data synchronization services is increasing day by day.In many scenarios such as data disaster recovery,data migration and data extraction,data synchronization is regarded as an indispensable technology.The highly reliable incremental data synchronization system introduced in this paper is an important part of the database infrastructure of Dareway Software Co.,ltd.,which provides data synchronization services for the company's smart government affairs,smart medical insurance and other businesses.In order to ensure the security of business information,this paper designs a complete and feasible technical scheme in each stage of data synchronization in combination with the business characteristics of enterprises.This system can support MySQL database and Oracle database in the data extraction stage.After extracting incremental data,the incremental data is stored in message middleware.The system uses transaction analysis technology to parse incremental data into database transaction objects.By analyzing transaction object information,the type of transaction can be judged and the transactions in specific types can be filtered according to business requirements.By analyzing the relationship between transaction objects,we can plan the correct transaction concurrent redo sequence,improve the efficiency of transaction redo,and effectively solve the performance bottleneck of data incremental synchronization system in data change redo.This system can accurately record the progress of data incremental synchronization,monitor the working condition of each link of the system,deal with uncertain factors such as network failure,storage failure,database meta-information error and so on,and fully guarantee the reliability of data synchronization.This system has been applied in smart medical insurance and social security systems in many cities.It runs smoothly and stably,and continuously provides highly reliable incremental data synchronization services.This paper introduces the background and related research of data synchronization,analyzes the functional and non-functional requirements of data synchronization,and introduces the design and implementation of a highly reliable data incremental synchronization system.The system mainly includes six modules:log processing module,transaction information storage module,transaction analysis module,transaction redo module,system management module and system monitoring module.This paper introduces the working mechanism of each functional module and how to ensure the service reliability.