Research Of Big Data Migration And Deployment Based On The Cloud Environment

In the era of Big data,data has been quietly affect our life,work and study,such as: social network,mobile client application,wearable devices which generated by the size of its produce to TB scale even PB level increasing trend.Cloud computing technology,as a new comprehensive technology,create unlimited value for information age development.At present,the government,enterprise software and data migration to the cloud computing environment,deploying and running also belong to one of the major requirements of national informatization construction.In this paper,combining with the relevant technology of cloud computing and Big data research towards the cloud data migration and deployment.First of all,this article in the introduction part mainly elaborates the research and development status of cloud computing,Big data and data migration.Then in the second chapter of this article in-depth analysis the basic theory and key technology related to cloud computing and Big data.In the third chapter of this article in detail based on Hadoop data migration and deployment of architectural design,and the thorough analysis in the data migration data partitioning module.Through in the third chapter,based on detailed analysis of the key problems in the process of Hadoop data migration,such as: the split-by value,num-mappers values,etc,for clues in the fourth chapter of this article designs the specific test cases,and several times of experiments.The fifth chapter of this paper in-depth analysis of the experimental results and the corresponding summary,by observing the analysis of experimental results,on the one hand,found in the process of data migration map task number(num-mappers value)is not the bigger the better,on the other hand found different data types of data partition of data migration efficiency and performance can also cause certain effect.Finally,this paper summarizes and planning for the future work.In a word,this paper mainly focused on traditional relational databases(RDBMS)between the cloud data migration performance issues.In this paper,data transfer module design is mainly based on Hadoop cloud platform,and the performance testing of the data migration tools Sqoop.Through the testing and verification of a large number of experiments,based on the Hadoop cloud platform of data migration on some properties is worthy of further research.This paper hope to make full use of the data of traditional technology and new types of large data processing technology,provide data analysis and processing of better performance and better efficiency.