Research And Implementation Of NVM Based Hybrid Key-Value Storage System

Clouds and mobile Internet have developed rapidly in recent years,and various applications have shown higher demands on the read/write performance and capacity of the storage layer.Non-volatile memory has comparable memory latency and bandwidth,and does not lose data due to power outages,meeting the needs of many low-latency applications.Key-value systems in NoSQL database are favored by researchers in the industry for their high performance,scalability,and flexibilityLog-Structure merge tree(LSM-tree)is a commonly used index structure of the KV storage system.According to its design,random writing which is not friendly to hardware device is replaced with sequence writing,and leads to greatly improvement on write bandwidth.However,the compactions of multiple layers introduces a large amount of additional reads and writes,which hinders the normal write process and increases the risk of flash device failure.Therefore,it is meaningful to do researching on K-V system based on NVM.Based on LSM-Tree,this paper proposes a hybrid index structure by applying the strategy of non-essential metadata separation.By dividing data into seperated range with different key prefix,the probability that keys duplicate with others is reduced,leading to high space utilization.With index data respectively managed in DRAM and NVM,impact of metadata updates on write speed reduces during heavy writing.Additionally,A coarse-grained memory management algorithm is designed for NVM.This algorithm considers the problem of uneven write load while reducing external fragments and ensuring memory allocation efficiency.A custom workload index is used to roughly estimate write workload of space.The memory allocating is according to workload index,and the index is updated if the allocating action triggers block merging.Gradually,workload distributes to all blocks and files almost equally.Based on open sources KV Database "LevelDB",HyKV is implemented and benchmarked on read/write performance and workload distribution.The results show that the HyKV system effectively improves performance compared to other common LSM-based key-value systems,and the space utilization is high,and the write load distribution is basically balanced.The sequential write performance is 1.7 times that of SSD-based RocksDB,and read performance is 1.2 times that of NVM-based RocksDB.the space utilization is over 90%,and the max gap of workload between mapping files is less than 10%.