Drilling Data Warehouse Etl Tool Research And Realization

The development of computer and the network technology made the enterprise to accumulate the massive data resources under each kind of application system, they constituted enterprise's precious wealth. Today, more and more enterprises are constructing the data warehouse to meet its strategic decision requirement, and must integrate the data possibly from different software and hardware platform. Therefore, provides one kind of ETL tool is the extremely beneficial work. That using the ETL tool extract data from isomerism data source and transforms, loads it to the warehouse, provided the high quality data for the decision analysis application that based on the data warehouse.Firstly, this paper introduces ETL process and the data warehouse, the ETL process include data extraction, transformation, cleaning and loading stages. I analyze the very important data mapping for ETL process. Then this paper analyze the traditional ETL tools for the architecture, according to poor opening and redeveloping, I raise the based on the metadata of the three-tier architecture, that make ETL process more efficient, versatile and flexible.In the research of heterogeneous data superaddition technology, I analyze the data superaddition technology basis of snapshot differential algorithms, the data superaddition technology basis of log check, the data superaddition technology basis of trigger and the data superaddition technology basis of timestamp.In data transform technology, this paper raises a kind of the based on the metadata transform methods. The data transform stage is separated, while provides data transform reuse mechanisms, who may save the rules and use its for extracting the daily incremental data, transform and loading; Users can also redefine data transform rules by their own needs. In this way, the flexibility and versatility of the ETL process is increased.Finally this paper according to the actual demand of the drilling data warehouse, I make use of the previous theoretical research results, design and implement the drilling data warehouse ETL tools. It is the based on the metadata of the three-tier architecture, The way that based on the metadata transform methods achieves that extracting, transforming, and loading the drilling business data source data into the data warehouse drilling. The tool includes five modules: data access module, metadata management module, data supperaddition module, task management module and data transform, loadable module. Users can make use of the task management module for the allocation of tasks and stores in the meta-database, and then temper timely the tasks. The ETL tool achieves the loading of daily incremental data by the data superaddition technology basis of timestamp; If the task requires be changed, Users can also re-configure the task. Therefore, the tool achieve its exclusive purpose, also has the flexibility.