Design And Implementation Of Etl Tools In The Data Integration Environment

The paper worked over the ETL tools of data integration, put emphasis on the ETL mode based on Web Services and Human Intelligence's application in data clean field.The paper firstly introduce ETL's conception and its'study status quo; and then describe briefly some key technology of ETL; in the third part, the paper show the detail design of some ETL's module; and it introduce key module's amelioration and realization in detail in its'fourth part.One characteristic of the paper is setting up the ETL based on Web Services and its metadata rule based on directness map to improve its'adaptability. Another characteristic of this paper is the application of Human Intelligence in the data cleaning process. The paper has revised the decision tree arithmetic to improve it's automatism, the multi-pass sorted-neighborhood and position coding arithmetic have also been revised and improved the detecting precision, effection and automatism of approximately duplicated records cleaning according to the experiment. Accordding to the detection of abnormal data, the paper classified the data firstly, and then detects the abnormal data based on the statistics theory and the operation rule database, it combines the results of detecting and get better effection. And the paper has set up the ETL Client module based on Ajax technology to improve the client's mutual experience.