An Etl Framework Based On Metadata Management

The past two decades are the golden age of the computer industry. Development of the Internet and the maturity of computer hardware and software, have resulted in the business data showing explosive growth. How to effectively manage a large number of heterogeneous data and integration of these heterogeneous data, and how to use the data generating decision making for the business management, becomes the problem which technical experts need to solve. Data warehouse is a kind of rising database application in recent years to address these issues.In fact, the data warehouse is an architecture, not a technology. One of the most important parts of DW is the metadata, because the metadata runs through the data warehouse project, which is the cornerstone of the data warehouse. On the other hand, one of the most important technologies used to build the DW is ETL, a kind of mechanism which changes the data state. As the data warehouse project differs from other software projects, in which business requirements often change. This directly leads to changes of the metadata and ETL process. Therefore, introducing the efficient metadata management model, and integrating it organically with ETL development, will take a multiplier effect in the actual data warehouse projects. That has become the main subject of this paper.Firstly this paper discusses the theories and methods involved in the subject, which refer to several major steps of data warehouse and the utility of tools. The paper introduces several types of metadata, their difference and specific business significance. Secondly the paper takes a brief review of the ETL technology, and an analysis of two basic methods to create ETL process and their advantages and disadvantages. Combining with the common warehouse metamodel (CWM), the paper then adopts a confederation-style metadata management system. Taking the advantage of the system in ETL development, we design a framework which can generate ETL process automatically. Finally we use the framework in our current project development, find that it can reduce the project cycle and improve efficiency. The result thereby verifies the feasibility of the framework.