A New IoT Storage System Based On Raw NVM

With the advent of the era of interconnection of everything,the importance of IoT(Internet of things)systems has increasingly prominent.The IoT systems usually contains numerous IoT devices and generates a large number of time series data continuously at a regular time interval,which presents a great challenge to how to store and manage these IoT time series data efficiently.NVM(Non-Volatile Memory)storage devices have the advantages of byteaddressable,low read-write latency,high read-write speed,large capacity and so on,which provides opportunities for efficient storage and management of large amounts of IoT time series data.Existing IoT time series data storage systems are usually re-developed based on general databases or storage engines.In the absence of a native storage mechanism for the characteristics of IoT time series data,there is also a lack of optimization strategies to adapt to the NVM storage devices,which makes it difficult to meet the requirements for efficient storage of large amounts of IoT time series data.Therefore,this thesis aims to study and design a new IoT storage system based on raw NVM.Firstly,according to the characteristics of NVM storage device,the existing IoT time series data storage systems are analyzed from four aspects: storage algorithm,query strategy,cache mechanism and consistency mechanism.The structure of the new IoT storage system based on raw NVM is given,which contains two main parts: the multi-granularity auto-converting IoT time series data management engine and the efficient caching and consistency mechanism.The storage and management efficiency of IoT time series data can be improved by effectively utilizing the advantages of NVM storage devices and the characteristics of IoT time series data.Secondly,according to the access characteristics of IoT time series data and the advantages of NVM storage device,a multi-granularity auto-converting IoT time series data management engine is designed.First,a multi-granularity auto-converting storage strategy for IoT time series data is presented.While decomposing the IoT time series data in time series,the IoT time series data is organized by using different granularities of data blocks,which reflects the timeliness of access to the IoT time series data,and also avoids the additional replication overhead when managing the time series data,thereby providing support for improving the storage and management efficiency of the IoT time series data.Secondly,a timeliness-based heterogeneous query strategy for IoT time series data is designed.Using the storage structure and access characteristics of IoT time series data,different query algorithms are designed for IoT time series data of different timeliness levels,which can improve the query efficiency of IoT time series data while avoiding the overhead of maintaining complex indexes.Finally,the prototype named TS-MGE,the multi-granularity auto-converting storage strategy for IoT time series data is implemented based on Intel Optane DC Persistent Memory and its storage device driver.Two appropriative testing tools for time series databases,YCSB-TS and TSBS to test and analyze respectively.Influx DB,Open TSDB and Timescale DB are used for comparison.The results show that TS-MGE can increase the write throughput by 137%,random query throughput by175.4%,scan throughput by 213.4%.Finally,an efficient caching and consistency mechanism is designed to solve the problems that the existing storage systems’ caching and consistency mechanisms cannot adapt to the characteristics of NVM storage devices and IoT time series data.First,a region-activated address prediction exception caching is given,the address of the correct data block is cached after the prediction-based query error,and the cache management algorithm is improved by using a region-activated method,which adapts to the storage structure and query characteristics of IoT time series data and improves the query throughput of low timeliness IoT time series data.Second,a fused outlier log is designed,which improves the query efficiency and management efficiency of outliers by converting outliers to the outliers summary.Then,a consistency strategy for IoT time series data storage is designed to ensure the consistency of data block writing to NVM and the consistency of auto-converting process.On the basis of Intel Optane DC Persistent Memory and its device driver PMEM,the prototype NBTSMS of the new IoT storage system based on raw NVM is implemented,and YCSB-TS is used for test and analysis.The results show that NBTSMS can improve the random query throughput by 180.1%,reduce the random query latency by 20%,reduce the log space overhead by 15%,reduce the outlier query latency by 48.1%,compared with Influx DB,Open TSDB,Timescale DB and TSMGE.In multithreaded tests,NBTSMS can increase write throughput by 103.1%,random query throughput by 135.7%,and range query throughput by 128.2%.