Design And Implementation Of Time Series Data Storage System In Kernel Space

With the rapid advancement of semiconductor technology and the storage industry,new storage devices such as NVMe SSDs continue to emerge,opening up new possibilities for the development of data storage technology.Compared with disks,new storage devices offer higher hardware bandwidth and lower I/O latency,drawing more and more attention to software overhead in the storage process,and the storage system's performance bottleneck has shifted from hardware latency to software overhead.In order to fully harness the hardware capabilities of these new high-performance storage devices,it is necessary to refactor traditional storage systems and improve them at the software level.As a result,reducing the storage system's software overhead is a subject worth investigating further.This thesis intends to address the problem of efficient storage of time series data such as video surveillance data in response to the ever-changing storage requirements of video surveillance data.According to the characteristics of video surveillance data and the drawbacks of traditional storage systems,this thesis proposes a distinct storage solution for video surveillance data and metadata.The software overhead in the process of video surveillance data storage is decreased by introducing a data storage model without trap called NTSM.And based on Rocks DB,it implements sequential writing and asynchronous data recovery of video surveillance data's two-level index data and other metadata.Based on this solution,this thesis designs and implements the time series data storage system in kernel space called NTSMSS,which adopts the Master/Slave architecture.Its data storage nodes use the block storage based on raw disk provided by NTSM to optimize the writing performance of video surveillance data and realize unified management of storage devices in user space.Because traditional storage systems rely on the operating system's system calls to complete the data storage process,NTSM implements message communication between user and kernel space by creating a cache-friendly SPSC ring queue pair based on shared memory,allowing user processes to efficiently complete data storage requirements without relying on system calls.Simultaneously,it prevents the mode switch caused by Trap frequently during the data storage process.To manage shared memory buffers,NTSM adopts a fixed-sized memory pool algorithm without loops,recursive processes,or additional memory occupation.File storage based on file system and block storage based on raw disk are both supported by NTSM.Among them,the block storage based on the raw disk directly bypasses the file system layer on the basis of obtaining the performance improvement without Trap and implements the zero-copy block storage process through the shared buffer technology,which dramatically minimizes software overhead.In addition,its kernel storage threads receive request responses from the block layer asynchronously to take advantage of those new storage devices' parallelism.It is well suited for the storage needs of massive time series data.Finally,the functionality and performance of NTSM and NTSMSS are tested in this thesis.The findings show that introducing NTSM can certainly reduce software overhead in the storage process and may bring more significant improvements with the development of new storage devices.