Video Shot Boundary Detection Based On Convolutional Neural Network

As an important pre-processing step for video semantic analysis and video manipulation tasks,shot boundary detection aims to identify abrupt transitions and gradual transitions of shots in a video sequence,and to segment the video into several shots automatically;how to identify the shot boundaries in a video accurately and efficiently is still a challenge.Current shot boundary detection methods are typically based on welldesigned hand-crafted features,the detection performance depends heavily on empirically determined parameters,and the scalability is not strong.These methods often design complex features,similarity measurements or classification methods to improve the detection accuracy,so they are computationally expensive and complex to implement.To tackle the issue above,a shot boundary detection model based on deep convolutional neural network is constructed.The whole model is divided into three stages.In the first stage,the candidate positions of shot transitions are located.The output of high layer of convolutional neural network is used as the feature representation of a video frame,then the dissimilarity between adjacent frames are calculated to quickly eliminate most of the non-transition frames.Lengths of gradual transitions vary greatly,so a video is downsampled with multiple temporal scales,and then candidate boundary frames obtained at different scales are merged together.In the second stage,a three-dimensional convolutional neural network is used to identify abrupt transitions in candidate frames,and the candidate position of gradual transitions are located based on the output of the network.The third stage further locate the time boundary of gradual transitions,another three-dimensional convolutional neural network is used to predict the probability of each frame being the start,middle and end of the gradual transition,then the start and end time of gradual transitions are determined by locating the strong peaks of the three probability signals.The model is trained and tested on public dataset ClipShots.The experiment results show that the detection model has good detection performance on both abrupt and gradual transitions;the performance of gradual transition detection is worse,and there is a lot of repeated computation.How to improve the performance of gradual transitions detection and reduce the computation cost remains to be further studied.