Design And Implementation Of Data Migration System Based On Hadoop Platform

In recent years,with the rapid development of computer technology and Internet technology,the era of big data has come,and in data application management,it is very important to improve the efficiency of data management system.But constantly expanded and enriched in the data type,data,and gradually developed into the unstructured data processing and storage objects such as reason,under the influence of the traditional database has lost the original control and dominance,unable to effectively solve the practical problems that exist in the data management,cannot be economic and effective to carry out data access,data analysis,data storage and so on a work.The application of Hadoop can improve the transparency of parallel processing and underlying storage,so that the database has high-performance storage capacity and cluster computing capacity.Subsequently,Hadoop becomes increasingly important in massive data processing and distributed computing.At the same time,applies the Hadoop data migration,also need to this platform and the relationship between the relational database to explore,learn how to improve the query efficiency,in the process of dealing with huge amounts of data to realize data migration,data import to Hadoop professional data analysis and processing,after the development for the key objectives of this studyThrough the collection and analysis of a large number of domestic and foreign references,master the current research status and application status of data migration system.To ensure that this research has a solid theoretical foundation,this research analyzed the Hadoop,cloud computing,the concept of data migration,and from two aspects of HDFS file system and graphs model introduces Hadoop technology system,after the Hadoop data query technology and the key technology of data migration has explored,such as Sqoop technology,ETL,etc.In order to optimize the data migration system based on Hadoop platform,this study designed the overall framework and data block model of the system.Then the steps to improve the data migration system are studied from two aspects of data partition and migration mode.In the process of implementing and testing the data migration system,the hardware configuration conditions,platform construction process,experimental process and so on were adjusted,and the data migration system was tested.Through the implementation and test of data migration system based on Hadoop platform,it is verified that the construction and application of this system can complete simultaneous processing of data query and data migration.Among them,the scheduling algorithm is improved in the process of data migration,and the basic performance of data migration is improved accordingly.In addition,combining Hadoop platform with data migration system and utilizing Hadoop platform to process big data can give full play to its efficiency in query.Therefore,this study has obvious reference value in big data migration and processing.