Research On Big Data Migration And Storage In Cloud Environment

In order to cope with the pressure of rapid data growth on online business operation platform,this paper designs and implements a big data migration storage system on the cloud platform.The system implements a data migration solution for enterprise daily operation management data and optimizes the data storage solution based on Hadoop platform.Existing business operations platform data is usually stored in a database of structured data,combining with the characteristics of structured data and storage environment,considering the migration efficiency,cost and other issues,this paper adopts the data migration based on Sqoop system design,and from the quantity and field data partition type,number of Map tasks three aspects to analyze the migration performance,this kind of design to the system in a concise and stable operation under the premise of fully take care of the security of data migration,at the same time to ensure the data reach the migration purpose after storage system can automatic archive.The Hadoop platform provides the HDFS distributed file storage system and the support of the MapReduce distributed computing framework.Based on this,the paper uses two data management components of Hadoop: HBase and Hive.Through the analysis of their respective advantages and disadvantages,complement each other,realize the data integration based on HBase and Hive storage system,and comparison analysis separately using Hive query performance and the use of integrated query system,the system can without destroying the integrity of the respective components function cases better play to the data query performance.After testing,the system can completely realize data migration,and the integrated storage system based on HBase and Hive can further optimize the data management performance of the big data platform.