Design On Logging Data Warehouse And Realization On Data Cleaning

To improve the management level of logging data and to overcome the disadvantage ofinconvenience preservation, easily damaged and low efficiency in the traditional log datamanagement, the logging companies and oil production factories have established loggingdatabases in succession. For the reason that the choice of the databases and databasemanagement systems made by logging companies and oil production factories may bedifferent, the characteristics of distributed, heterogeneous and sharing difficultly make itrestricts the utilization efficiency of logging data and the successful exploration anddevelopment of oil and gas field. Therefore, the design of logging data warehouse is putforwarded, which can provide the best information utilization solution. In this paper, theconstruction on logging data warehouse and realization on data cleaning based on datawarehouse are studied, the main works are shown as follows:(1) Study on the commonly data warehouse and data cleaning technology. On the basisof data warehouse technology analysis, the incremental data extraction method and the ETLdevelopment tool of SQL Server2005are determined to use, and the metadata databasesincluding sources of data, cleaning rules, analysis methods, etc. is built, according to the dataquality problem appeared in the pattern layer and the instance layer, different data cleaningmethods are put into used, and the data cleaning operation of the pattern layer and theinstance layer using the SQL order and SQL programming are realized. The experimentalresults show that the data cleaning methods are effective.(2) Design and development on logging data warehouse. According to the characteristicsof logging data and condition of varies logging databases, a design scheme of logging datawarehouse is put forwarded based on the existing data warehouse, and the development work of the logging data warehouse is completed. It can be divided to four layers in accordancewith data flow direction, such as routine database, underlying database, subject database andapplication system. The dimension tables and fact tables are built by use of the star datamodel, so as to realize the data integration based on file and database.(3) Application on logging data warehouse. According to the logging data warehouse, theWeb intelligent analysis and data mining application are carried out. On the aspect of Webapplication, query and download of logging data and intelligent operations on oil and gasproduction like proportion analysis, comparative analysis, trend analysis, etc can beexecutived by users. On the aspect of data mining, the self developed intelligent data miningsystem by linking the logging data warehouse can be used in logging interpretation such as oilgas recognition. Actual applications show that the logging data warehouse is scientific andreasonable in design, and can meet the application requirements.