Design And Implementation Of A Metadata Management Subsystem For Distributed Block File System

With the continuous expansion of the city's video surveillance system,security video surveillance system technology continues to advance,the traditional file system can no longer meet the storage needs of massive surveillance video streaming data.In order to solve the problem of storage and retrieval of city-level video data,video surveillance storage systems have begun to adopt widely distributed cloud storage solutions.Compared with a stand-alone file system,a distributed storage system that combines lowcost,high-performance,and easy-to-expand features undoubtedly has a great advantage.The storage system for massive amounts of high-definition surveillance video streams is write-intensive.The system must provide uninterrupted write services.A large amount of metadata operations imposes extremely high requirements on system performance,security,and concurrency.Therefore,the metadata management system plays an important role in the entire distributed system.This paper designs and implements a metadata management subsystem based on the DBFS,a distributed block file system developed for high-definition surveillance video streaming.The details are as follows:1.In terms of data storage,the system uses a logical volume structure to convert unstructured video stream data into block data for storage.This scheme effectively improves the write bandwidth of the disk and avoids the occurrence of a large number of disk fragments due to random disk reads and writes,resulting in a decrease in storage performance.2.In terms of data reliability,the single metadata server node is expanded into a cluster form,effectively avoiding single points of failure.The metadata server cluster adopts the Raft protocol for coordination.By selecting the master operation,log copying,and other functions,the state synchronization and data synchronization among the servers in the cluster are implemented to ensure the consistency of the metadata in the cluster.Effectively enhances the security and robustness of the metadata management subsystem;3.In terms of system scheduling,the system uses a self-adaptive load balancing strategy to target specific characteristics of video stream data.It first predicts the storage space required for each task,and then uses the remaining space of the disk.For comparison,select the node with the smallest ratio to issue the task.Through this scheduling strategy,load balancing among data storage cluster nodes can be guaranteed.The functional test results of the DBFS metadata management subsystem show that the system can provide the management functions of volume metadata and storage node metadata,and also meet the need for the metadata management subsystem to effectively schedule data storage nodes.It also achieves the design goals of system internal data consistency and system high availability.The performance test measures the performance of the system through the two indicators of network I/O and CPU utilization.The results show that the system has higher performance.