Research On The Architecture Of Big Data Platform Based On Video And Image

Along with the development of traditional and mobile Internet, large scale documents, images and videos appear on the Internet. At present, the amount of image and video data on the Internet has reached PB lever and it's growing rapidly, which indicates that we have entered the era of massive image and video data. In the face of such huge amounts of image and video data, traditional platforms can't deal with the data any more. Meanwhile traditional databases are disable to store this kind of unstructured image and video data well. Under this circumstance, how to build an efficient and easily-extensible image and video processing platform to meet the demand of storing and managing the huge amounts of image and video data, providing the service of processing images and videos becomes the key point.Facing this issue, this thesis designs a big data processing platform for images and videos. The platform provides functions to process the huge amounts of image and video data, including storing, computing and retrieving. In this platform, store and retrieve images and videos data on the basis of the data features. Hadoop and HBase make up this platform. Hadoop implements the distributed parallel computation to process the data, and HBase implements distributed storage of huge amounts of image and video data. For image data, use SIFT algorithm to get the descriptions as the features of the images and then transform the descriptions to indexes by using Locality-Sensitive Hashing. Map the similar feature descriptions that have the same index into the same hash bucket and store the image data. For video data, use color histogram to split video into video shots and extract the keyframes from the video shot. Then process the keyframes as the way of image processing and store relevant data and information. When retrieving data, extract the features from image and video data, build indexes of the features, and retrieve the data in HBase tales according to the index and return the most similar N images or videos as result to users.This big data processing platform is built on the foundation of master-slave architecture, which makes the the platform has good expansibility and fault tolerance. In this thesis, use experiments to test the performance of the platform through processing image and video data. The results of the experiments indicate that the platform has overcome the shortcomings of traditional platforms and it can process huge amounts of image and video data efficiently. The results also indicate that the platform can quickly and efficiently return the relevant data to users by retrieving the huge amoumts of image and video data.