Research On Distributed Reading Model Of Video Frame Based On HDFS

Facing the growing demand for mass video data processing,more and more enterprises choose to use HDFS such a distributed file system storing video file,Domestic and foreign scholars focus on how to read video frame data distributed and efficient when these frames stored in HDFS.However,there are three problems in the traditional video analysis application and video process library in distributed reading video frame operation: the distributed readability problem of video frame,the problem of inter-frame relationship maintenance and the problem of "data localization".In this paper,we proposed two kinds of video frame distributed reading models based on HDFS.According to the storage characteristics of HDFS and the particularity of video file structure and the characteristics of video frame operation.The work done in this paper is as follows:1)This paper presents a VSF-based video frame distributed reading model.In this model,we design VSF as an alternative to video files.VSF is a binary serialized file,the basic storage unit of VSF is Record.We form the video frame data which is decoded and frame number as a Key-Value pair and stored in VSF Record.A variety of Record filling strategies are designed for "data localization" and different types of video analysis tasks.In a distributed reading video frame operation,the application can directly read the decoded video frame data in the Record.2)This paper presents a distributed reading model for video frames based on VMD.In this model,this paper adds VMD components and video metadata files on the basis of HDFS cluster.The VMD component records the mapping of all the video files’ URL and their fingerprint summary information,and the video metadata file is a binary serialized file that stores the basic meta information of the video file and the I-frame dictionary of the video file.In the distributed reading model,the application obtains the video metadata file from the VMD component,fetches the file offset and frame number of all I frames using the I frame dictionary of the video metadata file,and the application can navigate directly to the I frame And sequentially reads and decodes the subsequent video frames.Finally,experiments were conducted on the availability and performance of the two models.In terms of usability,the two kinds of video frame distributed reading models proposed in this paper can be distributed to read and correctly decode the video frame data stored in HDFS.In terms of performance,the VSF-based video frame distributed reading model generates a large amount of additional space-time overhead in the VSF generation phase,while the video metadata file of the VMD-based video frame distributed reading model is smaller in size.The performance of the distributed reading model based on VSF is slightly better than that of VMD-based distributed reading model in terms of the performance of distributed reading video frame operation.