Research And Application In Real-time Analysis Of Massive Data

In recent years, with the high-speed development of information technology, it has become an important research direction that how to improve the data processing efficiency of massive data and real-time system. In systems which commonly using disk databases, it costs a lot of time in disk operations such as data processing, update operation and the like. To solve this problem, a better direction is to use memory mechanism, which can increase the speed of data processing and then achieve the real-time requirements of the system without destroy the original data structure. On the other hand, with the development of hardware materials and the development of computer technology, memory become cheap and the operation system can provide huge address space, which make it possible that all the massive data can be loaded into memory. In order to use memory mechanism to solve the problem of real-time analysis of massive data, this paper will explore the memory database. This topic of the paper is brought out from the "Research on key technologies of safety trusted telecom-level operation supporting architecture on reproductive health services" project, and is based on the decision support system, mainly to solve the slow processing speed of the workload statistics module.Firstly, this paper studied the theory of the memory database system, including memory database structure, the creation, allocation, management of storage space, two kinds of data organization and three indexing mechanism. Then, aiming at the differences in speed between the CPU cache and memory cache memory database, this paper studied the cache optimization techniques and analyzed the problem of cache mismatch ratio. Further, the paper compared the cache efficiency of several index structure and then designed and implement a hash index-based cache optimization method, and tested it to prove that it did improve main memory database system’s efficiency. Finally, based on the decision support system of "....", this paper designed and achieved an in-memory database system. It loaded the data from disk database into main memory, and managed its storage and provided index optimization to meet the needs of the upper business applications access.