The Research And Development Of ETL Module For Data Warehouse In Industrial Processes

The real-time/historical database has built plentiful production process data resources platform for the electric power enterprise, and it has important research significance and application value mining and the applying the information embedded in these data. It is the most effective method mining and organizing data, adopting technology of data warehouse. However, in the process of establishing the data warehouse, extraction, transformation and loading (ETL) is the momentous foundation, accounting for about 80% of the workload of building the data warehouse. At present, most large-scale relational databases supporting data warehouse have corresponding ETL modules, but they don't support processing the data of real-time &historical database. Besides, the production process data has its own characteristics, the processing method of which is completely different from the one of ordinary data. So, it is imperative to develop the ETL modules based on the production process. This article, firstly, introduced the concepts and characteristics of data warehouse, and the main functions of ETL modules. Then, for the characteristics of data based on production process, this article researched and provided some key algorithms for the historical data's cleaning, conversion, and extraction. Based on that, ETL application function software module was designed and developed for real-time& historical database. Finally, the ETL module developed in this paper was applied in building the data warehouse of analysis of operation condition for a 600MW coal-fired unit, and some results was also listed.