The Increment Data Processing Mechanism Based On The Materialized View Log

In recent years, Data warehouse is developing rapidly as data storage and management technology, it is known for its high level of data integration and data processing ability and other characteristics, data warehouse technology has received the widespread attention and the favor of many enterprises. Since the advent of data warehouse, it greatly promote the data mining and decision support in the areas of further technical optimization, it also can make the data have better integration, stability, and can be better to extract valuable information from the expansion of the data, and through decision analysis, data warehouse technology can make the enterprise gain greater benefits. Therefore, the study of key technologies in data warehouse will make it more conducive to business in the production of breakthrough, and bring more returns, it also make the data warehouse technology be applied to the business in all areas of production.The research of this paper-the increment data processing mechanism based on the materialized view log is on the basis of building a data warehouse, it is a key part of the maintenance data warehouse to keep incremental updating. This paper analyzes the domestic and foreign research status of data warehouse, and then studied the key techniques of building data warehouse, after cooperated and analyzed the incremental data acquisition and updating methods, proposes the DB-ODS-DW three-tier architecture and incremental data processing system based on materialized view log. The algorithm and the related code implementation of incremental processing mechanism was designed and implemented in the process of Thesis research, and then applied to the electric power company in the construction of data warehouse, this paper introduces the architecture of electric power company data center project and the key technology of data warehouse implementation. Finally, test and tune the performance of the system, in order to improve system’s efficiency.The paper proposed the DB-ODS-DW three layer architecture and incremental processing mechanism which can build corporate data warehouse and complete the data warehouse in the whole process of incremental data extract, transform, and load, and can shorten the time of the large data warehouse updated effectively, improve the efficiency and stability of a system. Especially for some enterprise data warehouse system which need daily update and rapid refresn and obtain incremental data, it may have more help. This mechanism can be well applied to the data warehouse technology in the industry field of decision analysis.