The Design And Implementation Of Data Incremental Synchronization System Based On MySQL Binlog

The rapid development of the Internet has promoted the transformation and upgrading of traditional industries.In the process of integration with the Internet,more and more enterprises have their own informational systems.However,due to technological selection and other reasons,it is hard to share data between these systems,which results in "Isolated Data Island".In addition,with the development of database technology,there have been various databases which are different from traditional relational databases,such as ElasticSearch and Redis.These databases have their own application scenarios and are often applied simultaneously in a system.It is a problem that many projects need to face and solve in development and maintenance how to share data between systems and synchronize data between databases.The data incremental synchronization system is a subsystem of an online marketing platform,for which it provides data synchronization service from MySQL to the report system,ElasticSearch and Redis.The system captures data changes by parsing MySQL Binlog.Compared to other methods such as using triggers,this method has less impact on the performance of MySQL and is more applicable to distributed systems.Then,the system publishes these data changes to the message oriented middleware.The consumers who need the changes of MySQL subscribe from the message oriented middleware and process the data changes.Load balancing,master-slave switching,reconnection and other mechanisms have been used to improve the performance,availability,and scalability of the system.The system has already been put into operation and works well,providing a stable data synchronization service for the online marketing platform.The system can be extended and applied to other projects that have data synchronization requirements.The thesis firstly introduces the background and development of data synchronization,and the main technologies used in the system.Then,it analyzes the requirements of the system,divides the system into four modules according to the results of the requirement analysis,including management module,synchronization module,monitoring module and consumption module,and introduces the interactions between modules and the design of each module.Finally,the code of key points and difficulties in the implementation are given with detailed descriptions.