Research And Application Of Massive Data Caching Algorithms And Design Patterns

With the rapid development of the Internet, securities, banking and other financial information system accumulated vast amounts of user data, the massive data and the rapid increase of the number of users bring a lot of pressure to these systems, how to effectively reduce user access latency and improve quality of system service is an urgent need to these information system. Web caching technology can greatly improve response speed of the system, traditional caching strategies, however, only focus on individual access to the user’s habits, not from a holistic cache performance. There are also some caching strategies based on the holistic characteristic, but they need to load all the data in into memory which is not possible for massive data. This paper focus on massive data caching strategies and how to design a caching system, the main research and contributions can be described as follows:(1) to eliminate redundant data for massive data system, we propose the REBDR (Rules Engine-Based Data Reduction) design pattern, using the rules engine as the data preprocess core, instead of the traditional data preprocessing logic curing in the code, so that the data processing logic and application code to obtain a good separation, the REBDR design pattern is versatile to different industries.(2) for the massive data system, it’s still impossible to load all data into cache after data reduction, we propose the data cache strategy based on the loading factor, the algorithm does not need all the data be loaded into memory, we can easily known if we should cache the data by the loading factor. In our simulation test, the loading factor caching strategy has higher hit rate compared to traditional caching strategy.(3) after applying the REBDR design pattern and the loading factor cache strategy in "e-banking risk monitoring system", we solve the most critical performance problems, the system has high flexibility and laid a solid foundation for domestic banks to carry out e-banking real-time risk monitoring.