Design And Implementation Of High Performance Cache Node Of Mass Stream Storage System

Nowadays,with the construction of smart city and development of smart transportation,a lot of cameras will be installed in cities and buildings.So how to efficiently process these massive video data will become a major difficulty in the construction of smart cities.In smart cities,there are many cameras and various types.This will cause huge data bandwidth,and due to various types of problems,it is likely to cause inconsistencies in video formats.This will cause huge difficulties for streaming data storage.So designing a high-throughput,low-latency,high-availability distributed stream data storage system will be the key core of the construction of a smart city.In the stream data storage system,the design of a high-performance buffer node will be the bottleneck of the entire system.How to design a high-performance buffer node for a streaming data storage system is the job of this article:1)Encapsulate the interaction protocol of stream data,and design interaction protocols with other types of nodes in the cluster.Consider the real-time nature of the video data.Here,the RTSP protocol is used to interact with the streaming data.In this experiment,the RTSP protocol is encapsulated.Processed all RTSP protocol messages.It also encapsulates the protocol interaction between the buffer node and the metadata node,and Implemented much functions such as registration and heartbeat.2)Optimized the buffering framework for receiving stream data.The buffer framework is the core design part of the buffer node.This paper designs a buffer framework that can deal with the real-time nature of streaming data.And another feature of streaming data is that the amount of data is large,so the consumption of memory must also be huge.Buffer nodes need to consider saving memory resources as much as possible.The buffer framework designed in this paper implements memory reuse,which not only solves the problem of saving memory.And reduce the overhead of memory allocation and memory destruction.3)Optimized the scheduling of read and write tasks.If the scheduling method is not changed,the traditional FIFO method is used to schedule the read and write tasks,which will inevitably cause unfriendly disk IO and reduce the read and write efficiency of the disk.4)A block management scheme for stream data is designed: The stream data of this system uses block management to manage the data.Each video block data is tagged with associated metadata.Each IO operation will operate the corresponding video block data through the metadata tag.