Multi-Level Feat Ure Fusion For Instant Mobile Video Search

In recent years,the rapid development of the mobile Internet has changed the way people record,watch and retrieve video.Mobile video search is a frontier research topic in video search communities.Although there have been much effort on mobile video search,it is observed that most of them rely on the traditional desktop video search methods.Besides,unlike traditional desktop video search,mobile video search also has its unique challenges:1)large aural-visual variance of query video;2)stringent memory and computation constraints on the mobile devices;3)network bandwidth limitation.In view of the above problems,this paper proposes a mobile video retrieval system based on multi-level feature fusion.The multi-modal hash frame is used to comprehensively retrieval the audio and video information about the video to reduce the influence of image deformation and audio noise on the retrieval accuracy.At the same time,the hash method is utilized to reduce the data amount of audio and video features,and the retrieval efficiency of the video is improved.Firstly,the paper uses the efficient convolutional neural network MobileNet to learn to extract the global semantic visual features,and uses the hand-designed algorithm to extract the local appearance visual features and audio features,and quantize the features into a compact hash code.Then in the online retrieval phase,the server receives by the above-mentioned various hash codes sent by the client,and performs video retrieval in the multi-layer retrieval index constructed.The method of this paper is experimentally verified on the data set built on the real scene.These videos in the query dataset are recorded by the mobile devices in the real world.The experimental results obtained 92.2%of the retrieval accuracy and an average of 733.2 milliseconds of retrieval time,achieving 2%higher accuracy than the-start-of-art and instant retrieval results.