The Research Of A Data Storage And Transfer Of HDFS Based On FTP Service

Internet's rapid development has brought mankind into the era of big data,with the integration of the Internet and traditional industries,the formation of the data business activities are increasing,resulting in massive data.But these vast amounts of data in different forms and scattered in various departments of the company,in order to deep mining and analysis of these data,the data integration,the migration and storage in a platform.For this reason,this paper studies the migration and storage of mass data,via FTP to achieve the migration of structured and unstructured data,using HDFS to store these data,we construct a data storage and transfer system of HDFS based on FTP service.In this paper,the FTP data storage and migration system based on HDFS service is designed and implemented in detail with the advantage of the Hadoop data platform architecture.Combined with the advantages of the hive and HBase.The integration of the two,design the storage system,the storage platform supporting MapReduce distributed computing framework programming,so as to improve and optimize the HDFS platform data storage performance.Considering from the aspects of security,efficiency and cost of migration,based on Java FTP service data cross platform migration,the migration process,data in form of file,the different types of data unified management and integration management.In order to test the system,according to the characteristics of the source data and the daily data management,three kinds of tasks are designed.Test results show that the system can efficiently complete the data migration and storage,with a certain rationality and practicality,so that the whole migration work and storage performance is optimized.