Design And Implementation Of Distributed Metadata Management System

In the wave of smart city construction,we have basically formed a city wide video monitoring system network in many cities across the country,and smart security and face identity authentication have been fully launched.The intelligent monitoring system based on face recognition technology needs to deal with huge picture stream data,so the existing system generally adopts the distributed system architecture.However,in such a framework of system,there are many nodes and a large number of metadata operations.How to ensure the consistency of metadata in the system so that the nodes can provide consistent services to the outside system is particularly important.Moreover,the system needs to deal with a large number of computing requests,and how to schedule reasonably according to the actual load condition of the physical machine so as to achieve load balancing is still prominent.In addition,under the complex conditions of hardware failure and network abnormality,node failure is inevitable in the distributed system.How to ensure data security and make the system have strong fault tolerance also needs to be solved.Therefore,how to provide a metadata management service on the distributed system to ensure the consistency,reliability and high throughput of system data is the key problem to be solved in this thesis.In this thesis,we design and implement a subsystem of the self-developed face recognition system,which is a distributed metadata management system.The theoretical basis,design scheme,implementation process and test analysis of the system are also described in detail.The main work of this thesis is as follows:1)In terms of data consistency,the metadata server node adopts the cluster management mode of one master and many slaves,upgrading a single metadata server control node to a server cluster of multiple homogeneous nodes,which solves the problem of single point of failure.At the same time,in the cluster,the primary node is selected to provide external services through the raft protocol,and the consistency of metadata in the cluster is guaranteed through log replication.Outside the cluster,different modules adopt time stamp strategy to ensure the consistency of metadata between systems.2)In the aspect of task scheduling,the improved weighted minimum connections task scheduling algorithm is adopted.For the task request of the client,the load host will collect the resource usage of the server under the load cluster regularly,and take the collected resource usage data as one of the weight factors to dynamically calculate the current load capacity of each server,so as to calculate the actual load capacity of each server more accurately,and improve the utilization rate of the whole system resources.3)In the aspect of data reliability,the data storage server adopts the strategy of multi replica redundancy to store multiple replicas on different physical nodes.When the data storage nodes fail,the system can automatically recover the data.This thesis implemented a lightweight distributed metadata management system.Through the functional test of the system,the metadata management system achieves the requirements of system data consistency and reliability.Through the performance test of the improved weighted minimum number of connections algorithm,the throughput of the system is effectively improved.