Research On Data Migration From Job Data To Job Data Center

At present, many large enterprises have constructed their own data center, because Data center can effectively integrate data and reduce the information isolated island. According to the Petrochina downhole data center as the background, focuses on the research of downhole operation in Daqing Oilfield Branch Company, these key issues are researched, such as the primary businesses, the data model construction of data center, data mapping meta-model and transferring model, and a data migration system is implemented,which can meet the business requirements.This paper mainly completed the following work:first, from drilling, workover, oilfield fracturing and maintenance of the four aspects of the downhole operation company operating data model is introduced, and the downhole operation company data center architecture is studied, from the development of fracturing, fracturing wells, well workover, acidification, fill hole completion, plugging water and production management and other functions of the seven domain, studied the date model and the date model dictionary of the downhole operation company.Second, and a multi-strategy data migration system framework is established, migration strategy defined three trategies, which includes historical data migration strategy, real time data migration strategy and instant data migration strategy. These strateges can be implemented by means of several metadata driven ETL model, which can finished the task of data migration of historical data and real-time data.Third,by the use of metadata and metadata driven theory and technology> a framework of data migration model is established, and, according to this framework, these meta-models are constructed, such as meta-model of the basicdata migration which is used to store customed metadata of mappkerin schema,meta-model of data migration which is used to store matadata of mapping schema.For the incremental data migration problem, a kind of shadow table method based on MD5algorithm is employed, the first ETL incremental extraction of traditional methods are researched, and meanwhile, these issues are researched too,such as, the trigger, timestamp, the comparison and contrast, log (mainly refers to the realization of the principle of synchronous CDC) and disadvantages of several methods, final all table comparison method is used.Finally, the above research results have been applied and verified in the downhole operation of Daqing oilfield company data center.The proposed data migration framework model is feasible and applicable, which can meet the demand of job data migration to data center.