The Design And Implementation For Storage And Distributed Processing Of Image And Video Based On Hadoop

In recent years,Internet and mobile Internet applications develops explosively under the impetus of Internet and mobile Internet technology.From large-scale portals to e-commerce sites,such as Sina,Taobao,Tencent,Jingdong and so on,which has brought great challenges to the technician while bringing the convenience to people's lives.People upload and share images and videos in every day,no matter when and where,they can do shopping online quickly and browse the information of goods.Images and videos gradually replace the text.The number of images and videos data stored in large sites almost reaches into terabytes daily and has a high concurrency access.Therefore,how to effectively store and process massive images and videos becomes particularly important.With the advent of Haoop,it has two core mechanisms: Hdfs and MapReduce,which open a new way of thinking for us.Shared storage of images and videos is implemented by Hdfs,users can read simultaneously data from multiple data nodes to solve the problem of high concurrent access.Map Reduce is used to realize the concurrent processing of data,which provides a favorable condition for the distributed processing of image and the distributed video transcoding.However,the original design of Hadoop is conducive to store large files instead of small image files,therefore image processing becomes the bottleneck of the Hadoop system.Aiming at this problem,this dissertation puts forward their solution through the study of existing solutions.This dissertation designs and implements the effective storage and distributed processing of images and videos based on Hadoop.The main work of this dissertation includes the following three parts:(1)Customize the realization of image storage optimization.Merge small images into large files,which can realize the addition of the large file.In addition,users can quickly find the location of image from large file by accessing the index mapping;(2)Rewrite the MapReduce input and output format interface to realize OpenCV's image parallelization in Hadoop,and take the image binarization as the case;(3)Achieve the parallelization of the Ffmpeg in Hadoop,and take video transcoding and adding watermark as the case.At last,this dissertation uses a performance comparison experiment to verify the effectiveness and necessity of image and video distributed processing.