Research On Surveillance Video Key Frame Extraction Based On Time-frequency Domain Analysis

In recent years,with the continuous improvement of people’s living standards and enhanced security awareness,the popularization of surveillance video systems has become the general trend.In video surveillance,the continuous work of massive monitoring equipment day and night has led to a sharp increase in video data.Quickly and accurately identifying and extracting key frames containing important content from massive monitoring data has become a research hotspot.While the existing key frame extraction algorithms have successfully extracted the target’s global state,they still need to be more accurate in capturing local detail information and suffer from compromised image integrity and lighting sensitivity.To this end,this paper breaks away from the conventional thinking and starts a series of technical studies on the existing problems from a time-frequency domain perspective.The main works completed are as follows.(1)A fractional Fourier transform-based key frame extraction method for surveillance video is proposed.A fractional Fourier transform-based key frame extraction method for surveillance video is proposed in response to poor extraction of target details and inaccurate judgment of local actions in the existing technology.Firstly,an order selection algorithm based on the golden mean point was designed to select the transform order and performs the fractional Fourier transform in this order to obtain the phase spectrum.Secondly,the mean square error of the phase spectrum of two adjacent frames is used to construct the feature expression of the target motion state change and form the mean square error curve.Finally,the local maxima of the feature representation curve are detected,and key frames are extracted accordingly.The experimental results on the public dataset validate the excellent performance of the method in extracting local detail information.(2)A key frame extraction method for surveillance video based on quaternion Fourier transform with multiple feature fusion is proposed.Aiming at the problem that the image multi-channel information needs to be fully utilized in the prior art,thus leading to poor extraction of detail information,a multifeature fusion surveillance video key frame extraction method based on quaternion Fourier transform is proposed.The method first extracts the motion features of the image,as well as the luminance features,red/green neuron features,and yellow/blue neuron features that characterize the RGB multi-channel information,respectively.Next,using a quaternion matrix,the fused phase spectrum is obtained by performing a quadratic Fourier transform on the proposed four features.Then,the fused phase spectrum containing information about the overall structure of the image is filtered by Gaussian filtering,and the quaternion Fourier inverse transform is applied to it to obtain the fused feature map.Finally,an adaptive key frame filtering criterion is constructed based on the average difference between adjacent frames of the fused feature map to facilitate accurate key frame extraction.The experimental results show that the average accuracy,average recall,and average F1 score of the proposed method are higher than90%,which is significantly better than the comparison method.(3)A key frame extraction method for surveillance video based on contourlet transform is proposed.A contourlet transform-based key frame extraction method for surveillance video is proposed for current methods that are sensitive to illumination and do not extract detailed information about the target direction.Firstly,a multi-scale multi-directional decomposition of the video sequence is performed using the contour wave transform to obtain an image with rich directional and contour information.Then,a combination of non-downsampling directional filters is proposed to obtain a non-photoresponsive contour feature map by filtering and fusing different directional features.Next,a texture enhancement model is constructed using a non-linear enhancement function to enhance the image edge information and the contrast of the target contours.Finally,a key frame filtering model is built based on structural similarity to extract key frames accurately.The experimental results show that the proposed method achieves an F1 score of 96% in the face of sudden changes in illumination and is resistant to photosensitivity.(4)An evaluation criterion for key frame extraction methods in surveillance video is proposed.To address the problem that the existing evaluation criteria for key frame technology cannot accurately evaluate key frame extraction techniques for the surveillance video domain,a key frame evaluation criterion based on the preservation of global and local motion information of the target is proposed is the target trajectory reconstruction degree(TTRD).Firstly,the key frames extracted by different methods are used to reconstruct the motion trajectory of the target in order to reproduce its motion state.Meanwhile,a quantitative analysis model is constructed to quantify the similarity between the original and reconstructed trajectories of the target to evaluate the performance of the key frame extraction technique.The experimental results show that the target trajectory reconstruction degree criterion can accurately describe the global and local variations of motion in surveillance video,which is more consistent with the visual perception of human eyes,verifying the correctness of the proposed method and demonstrating the effectiveness of the TTRD criterion in evaluating key frame extraction methods for surveillance video.