Design And Implementation Of A Distributed Video Stream Storage System

The widely use of video surveillance system provides technical support for the safety and development of city. With widely deployment of large scale video surveillance system such as "Safe City", how to store video data effectively and reliably become an urgent problem of the systems. Most traditional video surveillance systems use a centralized architecture, storing data with dedicated devices which connects multiple cameras. In case of massive high definition video data, traditional storage systems are facing performance, storage capacity, data reliability and system scalability issues. As a result, applying distributed storage system for video surveillance data storage become the trend in recent years.The author and the team he belongs design and implement a distributed storage system for video streams based on the study and research on distibuted storage, file system, coding theory and related technology. The contents of this paper are focused on the author’s work on system design and implementation and test of the storage subsystem.The author designs a data storage framework based on erasure coding, by replacing multiple replication in video stream storage system with erasure coding mechanism and data placement strategy within multiple nodes, the storage space cost is reduced in the condition of providing reliability in the failure of disk and node. Besides, a scheduling mechanism of storage node is proposed to minimize the amount of working nodes and reduce energy consumption on the premise that satisfying the writing performance and fault tolerance. In addition, the random writes of disk are transformed to sequential write through a data buffering strategy and modulation scheme of multiple concurrent streams, which improves I/O throughput as well as reducing disk loss. The metadata of the whole cluster can be managed by a single node as a result of effective metadata management mechanism, providing satisfying scalability, which enables the storage system to scale flexibly by simply adding or removing nodes.At last, the author did functional and performance test of the storage system. The results of functional test indicate that the system can meet the video stream data storage needs, providing data storage, reading and fault tolerance. Based on the analyzing of result of the performance test, the author believes that the throughput and storage space efficiency can satisfy the requests basically, and proposes several possible improving plans.