| Data synchronization is a must when a software system grows.Data needs to be synced to other isomorphic data store for redundancy,or needs to be synced to other heterogeneous data center for services.A 'Eventual Consistent' data synchronization has many different applications:Database admins need to use it for database real-time backup,and do data recovery when it needs;Microservice developers need to do data redundancy to reduce outer dependency or Join two table for speeding up query;Big data developers may need sync data from business database for large scale offline/real-time computing.Even though there exists some open source data synchronization tool,they do not make any 'Consistency' related promise and can't support some very complicated business situation.This thesis first introduced some basic theory and technique about'Consistency'.It begins with FLP impossibility,then CAP theorem,Eventual consistency,State machine replication,and finishes with 'At least once semantic' vs 'Exactly once semantic'.Some used technique includes 'Master Slave Replication Protocol','Write Ahead Log','Checkpoint','Retry','Order Scheduler'and'Idempotence'.In order to fulfill the requirement of eventual consistency,this thesis provides a suite of design.In aspect of capturing change of data,this tool adopts the'Master Slave Replication Protocol'of MySQL:it pretends to be the slave of master database and listens change from master;It also uses the'Write Ahead Log','Checkpoint','Retry' etc.,to make sure the eventual consistency.'Order scheduler' increases the concurrency level and makes sure the right order between change events.The data change capture procedure is designed as the 'Producer Module',and data change consumption is designed as 'Consumer Module'.The tool implemented as this thesis described has both good performance and consistency promise,which is proved in production.Now,this tool,Syncer,which is designed and implemented by me,is open sourced in Github. |
PDF Full Text Request