Design And Implementation Of Configuration-based Cache Synchronization System

Nowadays,cache is widely used in many backend systems.The synchronization mechanism is required to ensure the consistency of the changing data between database and cache.In the traditional cache mechanism,the consistency is usually maintained by the user.However,there is no unified cache synchronization strategy between each team,and the link between business code and synchronized cache operations usually leads to inconsistency.At the same time,some business that rely on the database change data to trigger mainly through scheduled tasks to access the latest data in real-time and sends it to message queue and relevant calculation services,which will bring high query pressure to the database.When encountering the peak of data changes,it is easy to miss data.In severe cases,which may cause called calculation services fusing and a large backlog of messages.Aiming at the above problems,this thesis relies on a project of an enterprise to designs and implements a configuration-based cache synchronization system.Through the data middleware Canal to monitor the table changes of database,developers only need to configure the cache information that needs to be synchronized,the basic information of the calculation service and the message subscription on the system platform.When database data changes,the system performs cache synchronization,message sending and calculation services calling by tasks according to the configuration information,the system also provides a message filtering method to reduce unnecessary messages and a message aggregation method to control the amount of calculated data.This system changes the calling relationship between the data and the user through configuration management,achieves the decoupling of the cache and the business code,and the stripping and encapsulation of the basic operations.Developer can maintain related configuration information by using a unified configuration template,reducing the complexity of code,solving the inconsistency between cache and database effectively.This thesis details the entire process of the design and implementation of a configuration-based cache synchronization system.Firstly,it analyzes the current problems in cache consistency and real-time data processing,gets the research goals of this thesis,and analyzes the research status at home and abroad,then briefly introduces the key technologies required in the project implementation process,include data middleware Canal,Quartz technology and Redis technology,etc.Next,it analyzes and summarizes the related business based on the database change data,gives the functional and non-functional requirements of the system,models and analyzes functional requirements by use-case diagram.Among them,highlighted the design of the configuration template.Subsequently,the overall architecture and functional module division of the system are given,include the concept,logic and table structure design of the database.The design of the four modules of message distribution,configuration,message processing,and calculation is described in detail through UML diagrams,the compensation mechanism is introduced in calculation module in consideration of the execution of calculation task timeout.Finally,the system is tested according to the requirements analysis,corresponding test cases are designed,and the test results are recorded and analyzed.After testing,the system has implemented configuration management,audit management and other functions,it can complete cache synchronization,message sending,and calculation task execution according to the configuration.All performance parameters meet the expected requirements and have good performance and reliability.This system has been put into use in the actual production environment of an enterprise,and provides services for a million-level commodity system.The daily change message processing volume exceeds 10 million,which meets the project requirements.