A High-performance Distributed Storage System For Large-scale High-definition Video Data

With the in-depth advance of the "Safety City" and other projects in recent years,video surveillance system becomes the key construction and its construction scale extends continuously.The growing number of cameras,higher quality monitoring pictures and longer video retention cycles have led to explosive growth in video surveillance data,and how to effectively store these massive video surveillance data has become an urgent issue in the storage systems in both academic and industry.Traditional storage schemes generally use a centralized storage architecture,i.e.,storing video data as files in common file system.In order to handle multi-channel high-definition video streams for a long time,the traditional storage solutions are facing with the problems in storage capacity,performance,data reliability and system scalability and other issues in the high concurrent write scenarios.In order to address these problems,this paper designs and implements a high performance distributed storage system for large-scale high-definition video data,called DVSS,by analyzing the characteristics of video surveillance data and traditional storage schemes.In order to improve the storage efficiency of video data,DVSS proposes a video data storage model based on Raw device,which leverages the buffer technology and the high concurrent write scheduling mechanism to convert many small random writes into large sequential write,significantly improving the disk I/O throughput.DVSS leverages a two-stage index strategy to manage metadata,which can be respectively managed by the state manager and storage servers,thus reducing the amount of metadata managed by the state managers and eliminating the performance bottlenecks.In the meanwhile,leveraging the GOP(Group of Pictures)structure as the basic unit of the video data index provides the second-level video retrieval accuracy.In addition,DVSS,as a distributed storage solution,enables linear expansion of system capacity and performance by adding storage nodes.DVSS also designs the available mechanisms for metadata services and video data,which increase the fault-tolerance ability and reliability of storage systems.Finally,the performance evaluation and comparative analysis of DVSS are conducted.The experimental results show that DVSS has good storage performance and can meet the storage requirements of large-scale high-definition video surveillance system.