| With the arrival of era of big data,the traditional solution of data storage and processing has been unable to meet the growing demand.And more and more data need to be migrated to the Hadoop platform for storage and processing.Data migration as an import research area in data science and technology,also concerned by more researchers in academia and industry.Existing data migration tools have characteristics such as pool performance based on single machine,complicated installation and program failures sometimes.In this paper,the shortcomings of existing tools,combined with the research has successfully Designed a Hadoop cluster's data migration services.Contribution in this paper are as follows.1)Design optimization based on database and log stream data extraction and migration.By parsing the database logging,incremental data extraction,and package t hedata directly into a message destined for a Hadoop cluster.Greatly reduce the 10 stream data extraction and network overhead.2)Factor analysis and response time are used to assess machines'load condition.Greatly improves the throughput of data migration system and cluster computing pow er.3)data migration cloud service design is able to better improve the overall transport system capacity and throughput.While migration task has a certain failure recoverability. |
PDF Full Text Request