Research And Application Of Data Synchronization Mechanism Based On Distributed Databases

With the rapid development of internet application, people have an increasingly high demand of information accuracy and promptness. Most enterprises store their information in the database system because it is acknowledged as the most convenient and efficient means to store data and retrieve information. The current trend is that database systems develop from miniaturization to large-scale ones and from centralized ones to distributed ones. In the distributed database system, to achieve synchronization of updated information from different databases lies in maintaining data consistency and integrity as data is stored on different nodes in the same network. In this context, data synchronization of the distributed database systems has become a hot topic.Based on data synchronization module of the project "Hubei Waterway Fees Levying System (Phase Two)" as well as the overall analysis of the network environment of sites on different levels of Port and Shipping Authority in Hubei Province, the data of the system can be divided into four levels, namely, Provincial Bureau data center, municipal, county, and primary sites. Initially, SQL replication technology was used to achieve the synchronization among more than 200 sites. The loss of data and untimely synchronic updating emerged during the trial. The thesis proposes effective improvements of SQL Server replication technology aiming at avoiding the data loss by reducing four levels to two levels, removing cascades, cleaning the related system tables and so on.In addition, the thesis designs a new synchronization mechanism by use of SQL Server transaction processing, ETL and XML technology. Firstly, the synchronization module makes use of SQL Server's transaction processing and data warehouse's ETL technology to construct a middle layer, so-called data acquisition system in the thesis, which collects incremental data needed to store them in the data warehouse, then, commits compression, transmission, decompression and other data processing. Data uploading and downloading can be fulfilled through XML technology, and the process goes like this:first, convert the stored data into the XML Schema documents so as to generate XML ones for uploading and then map the XML file to the target servers for downloading.New ETL and XML-based synchronization mechanism is mainly proposed by analyzing the instability of network environment and data overloading while ensuring data consistency and integrity by introducing new technology. Compared with the former replication mechanism of SQL Server, it has some obvious advantages. The ETL technology effectively alleviates the server pressure and reduces the network load. The XML technology is efficient, reliable, and suitable for mass data processing. And it can be extended to heterogeneous database synchronization.