Design And Implementation Of The Agent System Of Redis Cluster

With the continuous development of the mobile Internet,the amount of user data generated by mobile apps has increased dramatically,and the data model has become more and more complex.In some application scenarios,structured data models based on MySQL and Oracle cannot meet business needs,and the read and write performance of structured databases can easily become a system bottleneck.How to effectively store unstructured data and reduce the pressure of reading and writing structured databases has become one of the problems that Internet companies must solve.The usual solution in the industry is to introduce distributed non-relational databases,provide storage and caching services,increase the storage capacity of the system through horizontal or vertical expansion,and improve the overall performance of the system.As a memory database,Redis puts all data in memory for management,which greatly improves read and write performance,and supports the persistent storage of multiple data structures,which effectively compensates for the defects of structured databases.In order to achieve that the implementation details of the distributed cache and storage system have nothing to do with the user,the operation and maintenance operation is transparent to the user,and the service is stable and reliable to the user,the design of the redis proxy system has important value.The project in this article is a project that the author participated in during the internship of ByteDance.The project is based on twemproxy for secondary development and mainly realizes proxy access to redis cluster.As a basic component within the company,redis cluster provides users with distributed caching and storage services.Considering the security and ease of use of the system,users cannot directly access the redis instances in the cluster,and must go through the middle agent layer software.This system has made a number of improvements on the basis of twemproxy to meet the needs of the actual production environment.I independently completed the entire process of project requirements analysis,design,development,testing,and deployment.At present,the system has been operating stably in the online environment,providing reliable services for a large number of business parties.In the process of completing the project,I first read the implementation source code of twemproxy and redis database,analyzed the relationship between data structures,and understood the software design architecture.Then according to user needs and system implementation requirements,the design and development of four functions were completed on the basis of twemproxy,including the proxy function for redis cluster,the client read and write load balancing function,the distributed ISC AN command function,and the cluster topology automatic discovery function..For the non-functional requirements of the system,the service discovery component and monitoring alarm system were designed and implemented.Finally,each functional module was tested and the expected goal was achieved.At the end of this article,a summary of the entire project and prospects for future work are summarized.