Design And Implementation Of Enterprise Heterogeneous Data Transmission Service System

With the rapid development of storage field,many types of storage have been derived.Different data storage types have their own characteristics.A system usually involves multiple types of storage,and also involves the flow of data between systems.This makes the demand for data transmission services more and more.In order to ensure the smooth migration of data from upstream to downstream,the corresponding data migration configuration system,scheduling system,as well as the corresponding high availability,high reliability and high-performance guarantee measures are needed.Based on this,I design and implement a heterogeneous data transmission system,which is divided into five modules: task configuration module,task scheduling module,task transmission module,monitoring module,meta information module.There is a certain connection between each module,and each module also has its own responsibilities.(1)Task configuration module: This module mainly provides the API interface for interaction with users and carries out parameter verification;through reading and writing the data of meta information module,it interacts with users and feeds back the user's operation content to task scheduling system.(2)Task scheduling module: This module is in the role of connecting the preceding and the following in the whole system.It not only schedules the task transmission module to execute the task by reading the configuration command left by the task configuration module in the meta information module,but also feeds back to the task configuration module by reading the operation process data of the task transmission system.(3)Task transfer module: This module is the core of this heterogeneous data transfer system,which is the specific module for data migration.First,it needs to pull the data in the upstream My SQL,then carry out data parsing,data filtering and data mapping,and finally write the converted data into the downstream Kafka.The module also needs to consider some factors,including availability,reliability and performance.(4)Monitoring module: This module is the foundation to ensure the stable operation of the whole heterogeneous data transmission system.Through the query of the meta information module,it obtains the running status of each module and each data transmission task.Based on this,it makes fault judgment and fault feedback to ensure that the system maintenance personnel can sense the fault at the first time and take remedial measures as soon as possible.(5)Meta information module: This module uses the open-source storage tools Mongo DB and Zookeeper for storage,which is an important tool for interaction between each module of the system.In this project,I design and implement all modules of the whole system,and through functional verification and non-functional verification to confirm that the final system meets the design expectations.The project has been online,has been used for business access,and meets the needs of users.This topic is of great significance to data expansion and subscription incremental change data and plays an important role in technical iteration of data storage industry.