Research On Distributed Processing Of Massive Video Data Based On Hadoop

With the development of Internet technology and technological progress, network video data is increasing dramatically, how to effectively store massive video data and efficient processing of massive video data has become the focus of the current research.Huge data volume and processing requirements make single-node storage and processing far enough to meet performance needs, the need for distributed technology to store and process massive video data. Therefore, based on the mass data storage and retrieval platform of distributed processing platform, this paper designs and realizes the distributed processing of mass data based on Hadoop. The research work mainly includes the following three points.Firstly, aiming at the characteristics of mass video data, the overall architecture of mass video storage and processing is designed. The system uses the distributed storage system (HDFS) to store massive video data, uses MapReduce architecture design target image feature extraction and feature matching calculation, uses the column database(HBase) to store the video summary data and so on through the distributed storage and the processing Based on Hadoop massive video data distributed processing system, this paper designs the distributed data processing system based on Hadoop from the mass video data storage and processing, combined with the system requirements and the characteristics of video data.Secondly, in the storage of massive video number, analyze the video data and storage requirements needed in the system processing, and design and realize the functions of video data storage, video digest storage and feature extraction. Aiming at the characteristics of video data, the storage method of video data table and video summary information is designed to be stored in HB98 database,and the video data is stored in the video database by using HDFS distributed storage system for high-throughput video data. Information,the system can quickly retrieve the video data,by extracting the eigenvalues, can be quickly processed. In this paper, the XML file is used to solve the difference between the platform and the application. Finally, the efficiency of the storage function of the system is verified by the data reading and writing experiment.Finally, in the aspect of video data calculation, three kinds of lifting system processing efficiency are designed: single point cutting method, distributed cutting method, frame distribution method, combined with stored video data and distributed processing system,The advantages and disadvantages of the cutting method and the experimental analysis of the three cutting methods are carried out. By comparing the experimental data, the optimal cutting method is obtained, and the more efficient processing method is selected,which is deployed on the Hadoop distributed processing platform. Data distributed processing, through a large number of experiments to verify the system in the storage performance, computing performance and other aspects of feasibility and efficiency, greatly improving the efficiency of video data processing,mass video data processing is the best solution.