| With the revolution of information technology in the past decades, data which is the core assets of government agencies and business companies gains more and more attention. But as data continue growing, the amount of data people faced become too huge to migrate using traditional techniques. The efficiency of data migration gradually attract the attention of relevant staff when they carry out application migration or data analysis.From the perspective of mass data migration, based on the existing data migration tools, this article builds a heterogeneous data migration system for massive data based on the real-time distribute stream processing system Apache Storm. This tool is composed of task splitting and distribution, and distributed data stream. The two parts cooperate with each other, and complete the entire data migration.The distributed data stream part consists of data extraction, transformation and data loading. Storm supply interface of Spout and Bolt component for its application. Spout component is in charge of extracting data from data sources, and bolt component is responsible for data transformation and data loading. Besides, through dynamic compiling scripts, this tool provide flexible custom data transformation.Task splitting and distribution is responsible for splitting task into smaller tasks, then these smaller task is distributed to data extraction node. A table of RDBMS to be migrated is a task. The first step of task splitting is to select a field of the table, and then split the table into some sections according to the values of the field. Each section corresponds to a sub-task. Task distribution is the process of sending the sub-tasks to Zookeeper, so data extraction node can request sub-tasks from Zookeeper.Besides, Zookeeper is used for high availability and high reliability between task dispatcher node and data extraction node when they need to exchange information. Through Zookeeper, this tool is able to register data extractor node and synchronize status of all kinds of nodes.At the end of this article, experiments show that the newly designed data migration tool fulfills the design goals and is efficient for huge data migration. |
