Cluster Monitoring System Memory Database Design And Applied Research

Since the 80s of the last century, technologies of main memory database systems have been developing rapidly and receiving increasing attentions in research and study. Applications growing in scales and complexities, especially web-based applications emerging widely in world, performance demands in back-end database systems are rising higher and higher. Since traditional databases manage data file on hard drive disks, the high latency of disk I/O operations impose significantly negative effects on overall performance of the whole system. Thus they have become increasingly inadequate to meet the ever-increasing performance requirements.With the sharp fall in semiconductor prices, it has already become possible to use large amount of memories to implement database systems (main memory database systems) instead of disks.By using memory as the main storage medium to avoid overheads caused by I/O operations between internal and external storage devices, main memory database systems significantly exceed the traditional disk-based database systems in overall performance, which provides new ideas and methods in implementing high-performance systems.However, due to the change in storage medium, main memory database systems differ from the traditional database systems in many aspects such as architecture, characteristics, design and implementation, which need to be re-considered and studied accordingly. On one hand, some of the data records in our monitoring system are inserted/deleted so frequently that traditional data organization methods (array, sec-seg, link) don't perform well due to high costs in (de)allocating memory spaces;on the other hand, the overall number of reading transactions is much greater than that of writing transactions in which situation traditional locking granularity is either so coarse as to lose concurrency, or so fine as to cost more computing.By introducing the development, characteristics, related conceptions and technologies of main memory database systems, combined with my practical project experience, this paper proposes the design and implementation of a maim memory database system, LeopardDB,which can be used in high-concurrency real-time cluster monitoring system, from the standing points of data organization, query optimization, concurrency control, transaction management and recovery technology. Also a high concurrency cluster monitoring system base on LeopardDB is introduced briefly. According to the characteristics that 1)some data records have frequent insertion/deletion and 2) the ratio between writing transaction number and reading transactions is relatively small, we improved and optimized the memory managing mechanism and locking protocol by introducing dynamic memory allocation balancing and double-granularity locking protocol in order to automatically balance memory space usage and boost transaction concurrency in our monitoring system. At the end of the paper, it demonstrates the application of LeopardDB in real-time environment by presenting the testing results.