Researsh Of Data Migration And Storage Based On Hadoop

The era of big data is coming, and the information stored by all of the enterprises units with ZB per day. More and more companies tend to multi-level data management system and transfer large amounts of data to professionall data management platform which is able to store, compute and manage large data. A data migration and storage system is designed and implemented in this paper. Enterprise can complete the routine tasks of data management, like data migration to the Hadoop platform and storage in the Hadoop platform through this system.There are two structures of data waiting for migration. The unstructured data stored in files and the structured data stored in database tables. Taking the security, the integrity, the efficiency and the costs into count, migration method based on FTP service developed is adopted at last after comparison and analysis of several methods. This method can ensure the safety of migration largely. At the same time, so that the entire system processes is more fluent and clearer. In order to achieve the automated data storage and archiving, parameter configuration documentation which explains how to handle the corresponding data file needs to be carried during the data migration.There will be a program in Hadoop platform accessing this documentation automatically and then executing the appropriate action based on the parameters like an interface of data processing in Hadoop. Hadoop is more mainstream big data platform architecture. HDFS is used to store and manage data files in Hadoop. Hadoop supports distributed computing framework MapReduce in programming. So Hadoop can provide secure and reliable data storage and efficient and flexible data calculations. Hive and HBase are the data management component based on Hadoop. They are essentially different and have their own strengths. In order to combine the strengths of both, I get Hive and HBase integrated achieving an efficient and flexible composite storage system.It is proved that the system is able to support the data migration and storage, while the integration of Hive and HBase makes the performance of the platform better.