With the decresing unit price, increasing capacity and processing ability of main memory device, the reaserch on main memory data management has attracted much attention in recent years, especially on the main memory indexing technology. However, main memory index has to rebuild the structure after the system crash and loss of whole data because of the volatility, which will waste too much resource and time. Moreover, the capacity of most main memory devices is much less than the secondary memory devices’, so it is very common that the main memory cannot hold the whole index structure when the scale of index is quite big. That is why most database applications always employ the traditional disk-based index to fulfill the high demand on stability, but the searching efficiency is still the bottleneck.We propose a series of methods for the problem of volatility and space restriction of main memory index, which can improve the reuseability of main memory index and the adaptability when the main memory cannot hold the whole index. The main contributions can be summarized as follows:(1) A durable CSB+-tree main memory index is proposed based on the memory mapping technology, which can transform the whole index structure between main memory and disk with high efficiency, without recreating the index when loading from disk, this will save much computing resource. For the memory fragment problem during the index updating, we propose a classified main memory management mechinisum to monitor and regulate. When there are too many memory fragments, we will call the rebuild operation to eliminate all the fragments based on the ordered key-value pairs.(2) Based on the durable CSB+-tree, we focus on the problem of memory space restriction. To improve the utilization of main memory in hybrid storing system and achive the win-win situation between space and efficiency, we propose a self-tuning durable index scheme-Tide-tree. Based on self-tuning algorithm, Tide-tree can allocate whole index into main memory and secondary memory adaptly based on the running environment and work load, which gurantee the completeness of index structure and achieve the optimized performance based on the available memory. To increase the independence of Tide-tree, we also propose the level address table(LAT) to manage each level separately and propose three kinds of efficient index query operations.(3) For the low efficiency of the secondary index on the distributed cloud computing platform HBase, we propose a self-tuning secondary index base on Tide-tree, and apply it to the traffic real-time data management scenario to verify the effectiveness of the proposed Tide-tree. We also propose the whole data partition framework, which involves an inserting operation based on temporal priority, a searching algorithm based on time interval partition and a storage-awared index migaration algorithm when too many index blocks are created in memory. |