Research And Implement Of Data Warehouse ETL Technology

The development of computer and the network technology made the enterprise to accumulate the massive data resources under each kind of application system, they constituted enterprise's precious wealth. Today, more and more enterprises are constructing the data warehouse to meet its strategic decision needs, and must integration the data possibly be come different software and hardware platform, data model and even in the geography distributes, in the management autonomous and in the pattern the isomerism data source. Therefore, provides one kind of ETL tool is the extremely beneficial work. Using the ETL tool extract data from isomerism data source service and transformation, loads it to the data warehouse , its main effect is cleaning up, standardization and compiles each kind of service data, provided the high grade data based on the data warehouse policy-making analysis application.This paper analyzed the domestic and foreign ETL tool research situation, also analyzed the present mainstream ETL tool structure, the characteristic, data conversion, data cleaning and the metadata. Base on the Neusoft Beijing public security information synthesis inquiry system, proposed one kind of more general ETL tool frame design model, in this background, used the Java language to realize ETL system which according to the different ETL process to carry on the nimble disposition .At present, most of the ETL tools transformation engine uses compilation script language to manger the complex ETL transformation, the operation is complex, specialization, not easy to use. This paper has introduced the DirectShow media file processing theory, put forward the ETL transformation graph concept. The ETL transformation graph is composed by certain function sole data processing units, eachprocessing unit needs combination and connection according to the different ETL process, form the data processing assembly line and complete the ETL process. The transformation graph process the complex ETL process nimbly.In the realization aspect, utilized the inherit and polymorphism characteristics of object-oriented language, utilized design pattern, made the system construction clear, well extension and flexibility.Finally test the system, the system run steady. The result indicated that the ETL tool use transformation graph theory designed can complete the data warehouse ETL process.