Implementation And Optimization Of Data Integration And Exchange System Based On ETL

With the in-depth development of information construction in all walks of life,it's an emergency to provide transverse data through the channel between each distributed applications.A data integration and exchange system is designed and implemented based on the technology of ETL.The key technology such as the ETL incremental exchange task model,ETL task scheduling method etc is studied and verified in the practical application project.Paper's main work and contributions are as follows:1)Design data integration and exchange system architecture,including the logical and physical architecture,and the overall architecture of each module detailed,For abnormal cause the failure of task in the process of ETL incremental data exchange,design method to discard repeating data in time window,to realize the ETL time window incremental exchange task model,reduce the abnormal effects on the efficiency of data exchange.2)Put forward a Scheduling Method for ETL Task Cluster,to optimize the allocation of ETL scheduling and execution process,to improve the utilization efficiency of computing resources.This method divides the ETL task scheduling and execution,according to the ETL tasks arguments,to implement batch automatically assigned,in the execution phase dynamic adjust task priority to optimize execution.Comparing the ETL single task execution,ETL task scheduling cluster expands the ETL execution ability.3)Implement ETL exchange business processes for the public security bureau battle command platform based on this system designation,deploy and run it.It has run for more than half a year online stably,in the application more than 100 ETL tasks are distributed on ETL task cluster execution machine,ensure that each task can get the chance to run,ensure the reliability of the timestamp incremental extraction data process,improve the efficiency of data extraction,verify the effectiveness of the system.