Research On Video Keyframe Extraction Technology Based On Visual Saliency

With the further development of Internet and multimedia technology,and the popularity of digital devices and mass storage devices,a large number of video information is generated every day.How to process and analyze video quickly and effectively has become an important problem to be solved urgently.Video key frame extraction is an effective method for fast browsing and retrieval of massive video data.It is also the basis of video summarization.Commonly used video key frame extraction methods include shot boundary method,image feature based method,clustering based method and motion information analysis method.These four methods do not make enough use of visual saliency,which results in the low degree of representation and redundancy of the extracted key frames,and low consistency with the key frames extracted by users.The purpose of this thesis is to extract key frames which are more consistent with human vision system and highly able to generalize the original video content.The main work includes two aspects:(1)A video key frame extraction method based on video saliency and twostep hierarchical clustering algorithm is proposed.The specific contents are as follows: The saliency detection algorithm is applied to the original video sequence.Combined with the underlying features and motion information of the salient region,a fusion feature vector is constructed.Then the redundant information is removed by calculating the similarity between the feature vectors.Secondly,a two hierarchical clustering algorithm based on mutual information is proposed.One time hierarchical clustering adaptively determines the clustering threshold,which is real time.Two time clustering artificial sets clustering threshold to meet the needs of different users.Finally,the frame with the maximum mutual information between the other frames in each cluster is selected as the key frame.Compared with the current mainstream video key frame extraction algorithm,the effectiveness and applicability of the proposed algorithm is verified,and the selected key frames are more suitable for human vision system.(2)A key frame extraction method based on PCA and temporal K-means clustering is proposed.The main research is how to use fast convergent clustering algorithm to cluster similar features on the basis of ideal feature extraction.The specific content of the work is as follows: Firstly,Principal component analysis(PCA)is used to extract the principal component of the fusion feature vector.The number of principal components of a video in time coordinates is determined by cumulative contribution rate,and consider the number of principal components as the number of key frames.Secondly,the similarity between video frames is calculated,and the distance time distribution curve is generated,and the initial boundary of clustering is determined according to the peak value of the curve.Thirdly,the initial cluster is optimized by the time series K-means clustering algorithm.Finally,the nearest frame of the distance cluster center is selected from each cluster as the key frame.The experimental results show that the key frame generated by this algorithm has a higher matching degree to the user's summary.And the content of the original video is highly generalized,and the redundancy rate is low,while the timing of the original video is maintained,the selected key frames are more suitable for human vision system.The number of key frames is determined flexibly according to the type and length of the video,which provides an effective basis for the user to extract the appropriate length of key frames.