A Highly Concurrent Time Series Solid State Storage System For IoT

With the rapid development of Internet of things(Io T)technology,the explosive growth of Io T time series data has put tremendous pressure on computer systems.The emerging non-volatile memory(Non-Volatile Memory,NVM)has many advantages like persistent storage,byte-addressable,read and write speed close to DRAM,etc.And can be used to build hybrid solid-state storage system with Flash-based SSD,which brings chance to computer system for solving the storage wall problem of Io T time series data.However,the current time-series database applications are designed for traditional storage devices,they lack the optimization mechanisms for NVM devices,so it is difficult to take the advantages of the NVM device whith these application.In addition,the existing time series data management systems mostly rely on the traditional database storage engines,they lack the native management mechanisms designed aiming at the characteristics of Io T time series data.Therefore,the purpose of this paper is to design a highly concurrent time series data storage system based on the hybrid solid-state storage constituting of NVM and SSD system for Io T.(1)We analyze the challenges brought by NVM devices to the current storage system and time series databases,depict the structure of the high-concurrency time series solid-state storage system for Io T,and design two main function modules,the Io T time series data management engine and the I/O engine.(2)We direct at the limitations of the existing time series data storage system's long I/O software stack and the lacking of native time series data storage management engine,design an embedded Io T time series data management engine.Based upon the intrinsic characteristics of Io T time series data and different features of NVM and SSD,a redundancy elimination and compression fusion strategy,a hierarchical management strategy and a heterogeneous time series data index are designed.Finally,the prototype of embedded Io T time series management engine named TS-NSM is implemented based on Intel Optane DC persistent memory and its open source deriver PMEM and SSD's driver NVME.Using YCSB-TS as the IOPS measuring tool.The results show that compared with Influx DB and Open TSDB,TS-NSM can improve the write IOPS up to 243.6 times and 174.3 times respectively and improve the read IOPS up to 10.1times and 14.4 times respectively.And we tested the compression algorithm performance of the prototype,the results show that compared with Influx DB,TS-NSM can save up to 29% of storage space.(3)We analyzed the read and write characteristics of Io T time series data,the limitation of NVM devices driver,then designed the multi-queue-concurrent I/O engine.In this engine,we proposed a multi-queue-based I/O management strategy by building multiple I/O queues and worker threads on the NVM driver,then we proposed a workloads balancing mechanism based on Io T collection frequency to further optimize the management of I/O requests in the multi-queues,balance the workloads on each I/O queue and improve the concurrent read and write abilities of Io T time series storage system.Finally,based on Intel Optane DC persistent memory and its open source driver PMEM,a prototype of multi-queue concurrent I/O engine,TS-Engine,was implemented.Using YCSB-TS as the performance measuring tool,the results show that TS-Engine can enable Open TSDB,Influx DB,TS-NSM's write IOPS increased by up to 0.6%,7.8%,18.6% respectively,and the read IOPS increased by up to 1.8%,0.9%and 10.6% respectively.The write latency of Open TSDB,Influx DB,and TS-NSM can be reduced by up to 0.9%,2.8% and 8.3% respectively,and read latency is reduced by up to 1.0%,1.4% and 6.4% respectively.(4)Based upon the embedded Io T time series data management engine and the multi-queue concurrent I/O engine,the prototype named Io T-HCTSS,of a highly concurrent time series solid state storage system for Io T was implemented.Using YCSB-TS as tool to test the IOPS of Io T-HCTSS in two scenes,changing the number of concurrent Io T devices and the number of concurrent read and write threads,and compared with Influx DB and Open TSDB both mounted with NOVA and EXT4 file systems.The results show that compared with Influx DB+EXT4,Open TSDB+EXT4,Influx DB+NOVA and Open TSDB+NOVA,with the increase in the number of concurrent Io T devices,the write IOPS of Io T-HCTSS can increase by up to 14.0 times,6.2 times,13.8 times and 6.0 times respectively.The read IOPS can increase by up to17.7 times,19.1 times,10.7 times and 17.9 times respectively.With the increase in the number of concurrent read and write threads,the write IOPS of Io T-HCTSS can increase by up to 236.9 times,164.4 times,229.9 times and 161.3 times respectively.The read IOPS can increase by up to 9.8 times,13.8 times,8.6 times and 12.5 times respectively.