Video Style Transfer Based On Temporal Consistency Constraint

Since the neural style transfer algorithm was proposed,related researches have emerged one after another,which has important requirements in image rendering,coloring,art creation and practical application.In video style transfer problem,it is found that the generated stylized video often appears ghosting,artifact,flickering,and the problem of low efficiency of algorithm which cost amount of time.Therefore,to study how to improve the temporal consistency of the stylized video and how to improve the speed of algorithm at the same time have important significance.The traditional video style transfer methods restrict the temporal consistency of stylized video through the positive and negative optical flow between adjacent frames.Although it can generate stylized video which obtains better visual effect,it takes a long time,and the speed of the algorithm is far from the requirements of actual application.While the existing video style transfer algorithm based on deep learning,which through feedforward convolution networks to stylized video,although stylized results can be generated in real time,however,the temporal continuity of the stylized video is low,and flickering phenomenon is quite serious.Therefore,this article mainly aims at studying video style transfer method based on deep learning and put forward the improvements.The main research points and contributions of this paper are as follows: 1.A video style transfer method based on optical flow constraint and feedforward convolutional neural network is proposed.In this paper,the traditional methods and the video style transfer methods based on deep learning is combined,using the ideas in traditional methods which used optical flow to constraint the video temporal consistency of adjacent frames,we also designed a temporal loss function to enhance the learning ability of network about temporal domain correlation between adjacent frames of video,thus improve the continuity of the stylized video and visual effects.At the same time,a video style transform network of single model and single output is designed.This network only needs to calculate optical flow data during training and can directly output stylized video during testing,realizing real-time video style transfer and significantly improving algorithm speed.This method combines the advantages of traditional methods and deep learning methods.It not only overcomes the flickering,ghosting and artifact problems in stylized video,but also improves the algorithm speed,and can generate video with temporal consistency and good visual effect.2.A video style transfer method based on long-short-term constraint is proposed.If video style transform network only uses the temporal domain information of adjacent frames during the training process,the long-term consistency of its stylized results may be poor for the two frames that are far from each other in the temporal dimension.In order to solve this problem,this paper put forward to add long-term consistency constraints in the loss network,redesign the objective function.And adding skip connections in the style transformation network structure,make the low-level features directly into high-level network,to enhance the learning ability of video style transform network in temporal correlation of videos,improve convergence speed of network,make the method be able to generate highly temporal consistent stylized videos.Considering that the deep learning problem usually needs tens of thousands of datasets,while the traditional optical flow method is timeconsuming and computation-intensive,which has strong limitations,this paper uses the optical flow network module based on deep learning instead of the traditional optical flow method to improve the processing speed of the training dataset.