Research And Implementation Of The Distributed Data Exchange Platform Based On Incremental ETL

Data Exchange Platform processes data exchange service between different databases and different data formats. It is to solve the problem that information and data can not transform freely between different application systems at present stage.After a systemic study on relate technologys which was adopted in current data exchange platform.Utilizing incremental ETL technology and distribution ideas, we establish distributed data exchange platform based on the incremental ETL which has been effectively applied in practical work. We specifically completed the following work:1,Proposed an new incremental data extraction method based on pivotal attributes comparison, which includes pivotal attributes extraction algorithm and incremental data extraction method model based on key attributes comparison, introduced the available scope of the method, compared with the whole table comparison method and the whole table deleting & inserting method, analyzed its space complexity and time complexity, demonstrated its superiority, verified the correctness of the analysis by experiments.2,Proposed a distributed incremental ETL optimization method based on data partitioning by row. On the basis of incremental data extraction, we have an improved the previous ETL optimization framework, proposed incremental ETL optimization framework; on the basis of same sequence work theory, we proposed mechanism of data partitioning by line as well as best value extraction method of the number of partitioning, and proves its correctness theoretically, then gives a specific algorithm, we also verified the accuracy of the algorithm by experiments and analyzed the experimental results.3,Proposed an overall framework of data exchange platform on the basis of theoretical study, and presented design and implementation of its key modules.Finally, the system was tested, while the results show that the system has excellent usability and stability.