Research And Implementation On Distributed Database Migration In Traffic Management Information Systems

With the increasing of tremendous data in traffic management information systems,it is necessary to switch from traditional Oracle databases to distributed databases.The first thing in the switching process is the migration of historical data.However,due to the continuous operations of the systems,it is necessary to decrease the program level transformation,reduce downtime,and improve the migration rate.The first problem is the massive heterogeneous data,and both historical data and real-time data need to be migrated.Therefore,the traditional migration scheme script execution failure is high and the transmission time is very long.Secondly,the data security and consistency cannot be guaranteed,the data is lost frequently during data transmission,and the transmission file is not encrypted,either.In the global view of the above problems,this paper proposes an online migration method that integrates heterogeneous data across heterogenous platforms,which includes the following keypoints,First of all,in view of the complexity of data structure conversion between heterogeneous data sources and the different structure of the same database object when migrating to different databases,this paper designs and develops a structure migration subsystem that supports reading the original database objects and automatically generates the corresponding distributed database objects,so as to ensure the smooth storage of different types of traffic management data in different distributed databases Profit conversion.Then,in view of the different heat of historical data and real-time data and the huge difference of storage volume,this paper designs and develops a data migration method that compresses files by compression algorithm,transmits historical data through FTP breakpoints and transmits real-time data through docker container,which ensures the transmission efficiency and data quality of different types of data.Secondly,in order to solve the problem that data is easy to be lost and tampered with in the process of data transmission,this paper designs and develops the data encryption function.Through MD5 encryption of files,data and fields,it ensures that the data will not be lost or tampered with in the process of data transmission.Then,aiming at the problem that traditional data migration can not track the current migration progress,and can not determine whether the total amount of data before and after transmission is consistent,this paper designs and develops a migration progress tracking program.By calculating the total amount of data transferred,transmission speed and expected completion time,the progress control of data migration is improved.Finally,the design,coding implementation and testing of the whole migration system are completed,and the data consistency verification between Oracle database and distributed database is carried out.The data consistency and accuracy of distributed database are verified successfully.Through this study,we have realized the migration of massive heterogeneous data to the big data distributed system in a short time,and ensured the data consistency.The research results have been successfully trial run in Guangdong traffic management information system and verified the feasibility.The research content can provide reference for the distributed data migration of the remaining provinces in China and other similar systems for data migration.