Design And Implementation Of Metadata Server For Mass Stream Data Storage System

With the development of video surveillance technology,all kinds of surveillance equipment have been all over the streets.Video surveillance technology greatly facilitates our life,but the storage of massive surveillance data also poses a severe challenge to the relevant infrastructure.Distributed storage systems can meet this demand very well.Compared with typical distributed storage system architectures at home and abroad,it is not difficult to find that high availability and metadata reliability of metadata servers are often the difficult points to be solved by such systems.This paper takes surveillance video stream storage as the background,and focuses on the research of metadata high reliability and metadata server high availability around the metadata server of the massive streaming data storage system.The main research work is as follows:1.Highly reliable metadata management model based on VBSM block equipment.The VBSM block device separates the metadata server from the persistent storage of metadata.The VBSM block device is used as a persistent storage carrier for metadata.The underlying distributed cluster provides multiple copies of fault tolerance to the metadata in the block device.The metadata server program is responsible for ensuring the correctness of the metadata in memory.The metadata in memory is persisted to the block device through regular refresh and immediate refresh.Compared with traditional storage systems that use native disks or databases as metadata storage,metadata reliability can be greatly improved;2.Multi-strategy dual-machine hot standby method.First,the fast / slow combined heartbeat mechanism is used to complete the active / standby negotiation,status monitoring,and server mode switching;In addition,the time synchronization mechanism with the database is used to achieve the second heartbeat detection of the running state to avoid the problem of split brain;3.Design and implement the core of mass streaming data storage system:metadata server.The metadata server program integrates five parts of functions:metadata management,task scheduling,data buffer node management,dual-system hot backup and fault detection,and task migration.And as the system administrator,it interact with other nodes(clients,data buffer nodes)in the storage system.4.In this paper,we test the function and reliability of the metadata server of the mass stream data storage system,and analyze the test results in detail.