Research On Some Problems Of System Software Based On Non-volatile Memory

With the advent of big data era,the data being collected,stored and processed has exploded extremely.The traditional ”computing centric” computer architecture mode,which with limited computation ability and scalability,has become more and more hard to deal with the exploding data volume.The existing computer architecture is facing serious challenges and survival bottleneck.Recent years,with the emerging of new-type non volatile memory(NVM)technology,because it possesses unique features including non-volatility,byte-addressability and high density,it brings huge challenges and opportunities to the existing computer architecture.To break the traditional architecture,building a large capacity,high reliable memory-storage heterogeneous system,holding all the data set in memory become an attractive model of in-memory computing technology.This change will cause the computing model transform from the ”computing centric” model to the ”data centric” model,and has become an attractive point in both academia and industry.This paper based on the non-volatile memory and the hybrid memory architecture as the research background,takes the storage system software as the main research target,discusses the design and implementation methods of storage system software which design for non-volatile memory.The research point in this paper mainly focuses on two systems,an NVM-aware file system and an NVM-aware key value store system.The content includes the design principle,implementation method,data consistency guarantee mechanism,and the experimental evaluation of each system.For the file system,We design and implement a NVM-aware file system named HMFS.In HMFS,we mainly discuss how a file system to manage the DRAM and NVM with a unified address space,how to put file system meta data and data on DRAM and NVM respectively and adopt different update mechanisms to them in order to achieve the trade off between system performance and medium endurance.We summary the different data consistency level in a file system and achieve the highest version consistency level in our file system by a novel multi-version data structure and a set of related algorithms.For the key-value store system,we design and implement a novel key-value store system named LibreKV.In LibreKV,we aim to reduce the storage software stack level,simplify the data consistency mechanism and enhance the memory utilization efficiency.In our key-value store system,we also construct our system on the hybrid memory architecture,use different hash table structures to achieve the balance between system performance and memory utilization efficiency.We adopt a checksum based consistency guarantee mechanism to avoid the expensive write-ordering ensurance primitives.The experimental results of LibreKV have shown that LibreKV achieved good scalability and excellent memory extension efficiency compare to the state-of-the-art key-value store systems.For summary,in this paper we illustrate the design and implement method of a hybrid memory file system HMFS and a hybrid key-value store system LibreKV independently.The main contributions of this thesis mainly include the following points.1.We design and implement a hybrid memory file system named HMFS.We construct HMFS on a DRAM and NVM hybrid memory architecture and make them share a unified address space.We adopt different update mechanism to file system meta data and data.For the meta data,we adopt the update-inplace update mechanism to achieve fast updating and simple management.For the data,we adopt the logstructured update mechanism to achieve evenly access to the NVM medium at most.We target HMFS as a full-versioning in-memory file system and use checkpointing technology and a set of versioning related data structure and version to guarantee the version consistency.We carried out a comprehensive experimental solution to evaluate HMFS from three different metrics: performance,versioning efficiency and endurance.The experimental result has shown that HMFS achieve comparable performance than other state-of-the-art file system and significant improvement on version efficiency and endurance.2.We propose a full-versioning approach with a set of data structure and related algorithms in HMFS.This approach not only fit for a in-memory file system but also fit for any other file system base on a logstructured layout.This full-versioning mechanism includes basic version related operations,such as version creating,version reading and version creating and a full versioning evolution principle,that is,any newcreating version can be born from all the existing versions in the file system,do not have to be born from the adjacent version,and a new version can has many parent versions.This versioning approach outperforms other existing file system with versioning and achieve smaller space overhead to store the meta data about versioning.3.We design and implement a hybrid in-memory key-value store system named LibreKV.We construct LibreKV on a DRAM and NVM hybrid memory architecture.In DRAM,we use static hash table to accept the IO requests from the client quickly.We perform cuckoo hashing in the static hash table to make the hash table achieve a high utilization.In NVM,we use a dynamic hash table and a dynamic extension algorithm to match the actual data size in the system dynamically.We adopt a checksum and checkpoint combined mechanism to guarantee the data consistency in LibreKV.This mechanism can avoid traditional consistency mechanism which uses costly write-ordering hardware primitives and can reduce the system overhead significantly.4.We propose a novel data consistency guarantee mechanism based on checksum technology.We try to adopt the checksum as the valid or invalid judge condition in the key-value store system and discuss the reliability both logically and experimentally.This mechanism takes good advantage of key-value store system's good hash calculating ability and avoid expensive hardware primitives to guarantee write-ordering.We adopt a checkpoint approach in the dynamic hash table located on NVM,this approach can reduce number of the key-value items which are need to be checked after a system crash or recovery.