Research On Change Data Capture Of The Heterogeneous Data Sources

In order to resolve the restriction of the information isolation and data sharing across different departments regionally, many city governments began to construct the unified government data centre, to support the information development of all departments. This is an inter-departmental trans-regional project. In the construction process of such project, it is a key issue how to solve heterogeneous mass data extraction, transformation and loading efficiency.So far in the market, the current heterogeneous data integration tools are not ideal solutions for massive heterogeneous data integration efficiency. The technology of Change Data Capture (CDC) is a good solution. In this project, we investigate the issue of heterogeneous changed data capture based on the "Public Service Information System", analyze and experiment the data characteristics and various technical strategies to find out an enhanced method to improve the massive data integration efficiency in the process of heterogeneous data integration.We first analyze the traditional techniques of the Heterogeneous Data Integration and Change Data Capture. Then by comparison, design and improvement, we implement the Change Data Capture on the Heterogeneous Data Sources based on a typical heterogeneous relational database platform. The main settlements in our study are: heterogeneous data sources under changes judgment, capture, transformation and loading; Remote connectivity to heterogeneous databases. Techniques such as Trigger-Based Capture, Transaction Log Capture and Change Trace Capture are applied to experiments according to the characteristics of massive data.This project is an experimental study of Change Data Capture of the Heterogeneous Data Sources based on the " Public Service Information System". By implementing the technique of CDC with the massive heterogeneous data, we can improve the inter-departmental trans-regional massive data efficiency in the development of e-government.