Research On Key Techniques Of Micro-video Content Analysis

With the rapid development of Internet technology and mobile devices,video acquisition,storage and propagation have been increasingly prevalent and convenient.Because of its integration of visual,audio and textural modalities,video has prominent ability in information expression.Hence,video becomes an important media form in sharing information on social network.As a new type of video,micro-video has very limited duration,which is usually several seconds.It greatly reduces the requirements in video production,propagation and viewing.Thus,the number of micro-video grows explosively on social network.The effective management and intelligent service for micro-videos emerge as the times require.Served as the foundation of effective video management and consumption,video content analysis techniques have made significant progresses in recent years.However,compared to conventional video,micro-video has some characteristics,such as various sources,high diversity,high information density,and sparse hashtag.These characteristics bring in new challenges to micro-video content analysis.In this dissertation,we investigate the inherent characteristics of micro-video and dedicate to studying several key techniques of micro-video content analysis,i.e.,object proposal,salient object detection,tag refinement and personalized recommendation.These techniques can provide strong support to effective management and consumption of micro-video on social network.The main contributions of the dissertation include:(1)We propose a novel object proposal method on micro-videos based on multi-feature fusion.Compared to object proposal on images,video object proposal is more difficult,because it cannot simply generate the candidates for objectness estimation,needs to analyze the objectness under different features,and requires to provide the bounding box trajectories with accurate locations and durations as the results.According to the characteristics of the objects in videos,we propose a video object proposal method based on multi-feature fusion,which can effectively propose the objects with different characteristics.Moreover,we improve the key procedures in video object proposal,such as keyframe object proposal and the generation and matching of short bounding box trajectories,by exploring the potential of spatio-temporal coherence and redundancy in micro-video content,in order to limit the computational cost.The experimental results show that the proposed method can effectively handle the micro-videos with complex content,and it obtains the better performance than the state-of-the-art video object proposal methods by considering both effectiveness and efficiency.(2)We propose a novel salient object detection method on micro-videos using spatial-temporal difference and coherence.Compared to salient object detection on images,video salient object detection should handle more challenges,such as detecting motion saliency cue,improving saliency coherence inter frames,and pursuing high efficiency,besides extracting static saliency cues intra each video frame.Based on the characteristics of salient objects in videos,we propose a salient object detection method by fusing spatial difference and temporal difference of video content,which can effectively detect both static and moving salient objects.Meanwhile,by making full use of the content redundancy of adjacent video frames,we only directly detect salient objects on keyframes and generate the saliency maps of non-keyframes by spatial-temporal coherence based saliency propagation,which can obviously improve the efficiency of the proposed method.The experimental results show that the proposed method can effectively handle the micro-videos contacting complex object motion and scenes,and it outperforms the state-of-the-art methods when taking account of both effectiveness and efficiency.(3)We propose a novel tag refinement method for micro-videos by learning from multiple data sources.Because the user-generated hashtags of micro-videos have seriously unbalanced distribution and low quality,it is challenging to directly describe micro-video content via hashtags.Meanwhile,considering the huge workload of constructing a large-scale,manually labelled micro-video dataset,it is also difficult to train an effective model for effective micro-video labeling.Considering the existence of many manually labeled datasets,which have relevant content to micro-videos,we propose a tag refinement method for micro-videos by learning from multiple public data sources with manually labelled tags,which are provided for object detection,activity detection and scene detection.The proposed method can overcome the difficulty of directly refining the imprecise hashtags and address the problem of lacking manually labelled micro-video datasets for training.It can obtain good performance in tag refinement of micro-videos with high quality or low quality hashtags even without hashtag.Moreover,we construct a micro-video dataset and manually label the microvideos with target tags.It can be used as a benchmark for micro-video tag refinement.(4)We propose a novel micro-video recommendation method based on hierarchical user interest modeling.Because micro-video has very limited duration,users prefer to obtain interesting micro-videos by automatic recommendation instead of selecting them manually.Effective micro-video recommendation technique should model user interest based on his/her viewing history and feedback,such as like and comment.Considering personal user interest and its changeability,we propose a hierarchical model to describe user interests in viewing micro-videos,which can describe both the long-term interests related to users' hobby and temporal interests in a certain viewing.Based on it,we present a micro-video recommendation method based on hierarchical user interest modeling,which can recommend micro-videos for different users according to their personalized and current interests.The proposed method can effectively model user interest in viewing micro-videos and it outperforms the state-ofthe-art methods in micro-video recommendation.