| With the rapid development of science,technology,and information engineering,the scale of data has grown exponentially.The data has infiltrated into various fields and become an important part of productive forces.The requirement of mining the latent value behind big data has effectively and quickly accelerated the development of data processing systems.The representive of the cloud computing platform,Hadoop,has become one of the important systems for big data processing.However,the data generated in the actual production activities are mostly concentrated in file systems or relational databases.How to import the data from the file system and relational database into the Hadoop platform's storage system and export the processing results from Hadoop to relational databases or external file systems have become urgent problems for many industries.In this thesis,the problem was addressed and a feasibile scheme is proposed on the basis of Sqoop.A data conversion platform based on Sqoop is implemented to support the parallel bidirectional conversion between the structured and semi-structured data and Hadoop.The application results show that our proposed platform is efficient.This thesis mainly makes the following contributions:1.Analyze and compare the existing conversion tools,and thus make a feasibile scheme of the data conversion platform.2.Based on Sqoop,a novel data convertsion platform is implemented.The main functions is as follows.(1)The platform improves Sqoop and adds the function of exporting data from HBase to the relational database in parallel.(2)Using the multi-thread technology and MapReduce,the platform implements the parallel bidirectional conversion between XML,CSV and HDFS,Hive,HBase.In order to meet the needs of various data sources in the enterprise,the platform provides a standardized interface for developing plug-ins.In this way,the user can customize the conversion rule according to their own data types.(3)The platform provides a graphical user interface and solves the defect that Sqoop only supports the command line.3.Our platform is effective and its performance is better than the traditional technologies. |
