An Automated Development And Deployment Framework For ETL Applications

The past two decades have been a blooming age of information technology. With increasing maturity of both the Internet and the hardware/software of computers under the Moore's Law, it has lead most industries into an informational and electric way of operating manner. Massive daily transactional information has been generated and stored in the processing systems and are even growing explosively as the days pass by. It has become an urgent research subject for IT experts on how to translate the vast information to human-sensible knowledge for business management, effectively. Business Intelligence (BI), Data Warehouse (DW) and Data Mining are concepts that being proposed under such scenarios.ETL (Extract, Transform and Load), is the heart and soul of a BI system. An ETL process usually extracts historical data from the daily business system, transforms them into some certain schema, and loads them to a Data Warehouse with pre-defined schema. The downstream systems can perform queries and analysis based on the DW.It is generally recognized that designing and developing ETL processes occupies 60 to 80 percents of a total BI project. It is of great significance to leverage the efficiency of such works.This paper provides two approaches that can improve the efficiency of carrying out ETL projects. On one hand, it designs and implements a framework that can help the developers with generating the ETL jobs from requirement documents, automatically. On the other hand, it researches on the deployment process of an ETL process. It discusses the current deployment process of ETL applications, analyzes the main shortages and finally shows the design and implementation of an automated deployment framework, which greatly curtails the time and reduces the complexity of the deployment process, also the entire development of ETL applications. It proves practical after applied in real projects.