Research On Video Frame Interpolation Algorithm Based On Global Optimal Optical Flow

The video frame interpolation takes continuous frames in the video as input,and uses the information of the input frames to predict and synthesize intermediate frames,thereby generating a video with a higher frame rate and a more coherent picture.As one of the hot research directions of computer vision,video interpolation has a wide range of applications in video frame rate conversion,video compression transmission,slow motion playback and other fields.Due to complex conditions such as occlusion,motion blur,and large-scale motion,the task of high-quality video frame interpolation is facing great challenges.Especially the complex and changeable motion modes in real scenes make it extremely difficult to perform accurate optical flow estimation.Therefore,the study of video interpolation in complex motion modes has important theoretical significance and practical value.This article focuses on the video frame interpolation algorithm under the nonuniform motion mode of the scene.By introducing the global optical flow of the video sequence,this paper proposes two optimization strategies based on polynomial fitting optimization and convolutional LSTM learning optimization,and constructs a video single-frame interpolation calculation model based on global optimization optical flow,and will be based on global optimization The video interpolation algorithm of optical flow is applied to the multi-frame interpolation model to realize the enhancement of the interpolation of the low frame rate video.The main research work of this article is summarized as follows:(1)A video single-frame interpolation algorithm based on polynomial fitting for global optimization of optical flow is proposed.Taking into account the limitations of the optical flow of adjacent frames in non-uniform motion estimation,this paper introduces the optical flow of the entire video sequence,and optimizes the global optical flow sequence through polynomial fitting and residual enhancement,so as to realize the optical flow prediction of the intermediate frame.Overcome the limitation of local information in the non-uniform motion mode.This algorithm is tested on the UCF-101 video data set.Compared with the base network that does not use the global optimization optical flow,the algorithm increases the SSIM of the video interpolation result by 0.0054,which verifies the effectiveness of the introduction of the global optical flow.Compared with Super-Slomo,this method improves the SSIM of the frame interpolation result by 0.0308,and compared with BMBC,the SSIM of the frame interpolation result of this method is only 0.0001 worse.(2)A video single-frame interpolation algorithm based on convolutional LSTM with global optimization of optical flow is proposed.Inspired by the convolutional recurrent neural network,deep learning is used to optimize the global optical flow.This paper proposes a global optical flow prediction module based on a convolutional LSTM network,which predicts the optical flow of intermediate frames through learning,and provides better optical flow estimation for interpolated images.The algorithm is tested on the UCF-101 video data set.Compared with the base network,this paper increases the PSNR of the video interpolation result by 0.05 db,and the SSIM increases by 0.0069.Compared with the first research work(based on polynomial fitting)This method increases the PSNR of the interpolated frame result by 0.08 db and SSIM by 0.0015,but the network parameters are increased by 0.6 times,and the time consumed for processing each frame image is doubled,which is comparable to Super-Slomo.Compared with BMBC,this method increases the PSNR of the frame interpolation result by 1.02 db,and the SSIM increases by 0.0323.Compared with BMBC,this method increases the SSIM result of the frame interpolation by 0.0014,and the PSNR difference is 0.02 db.(3)For low frame rate video,combining the above two algorithms,a video multiframe interpolation algorithm based on global optimization of optical flow is realized.The algorithm introduces time information,fused the time information t and the global optical flow prediction module to realize the frame interpolation at any time t,and realizes the multi-frame interpolation between two frames by simultaneously interpolating the video at different times t.The algorithm was tested on the UCF-101 video data set.Compared with the base network,the video multi-frame interpolation algorithm based on convolutional LSTM improves the PSNR of the video interpolation result by 0.04 db and SSIM by 0.0089.The video multi-frame interpolation algorithm based on polynomial fitting improves the SSIM of the video interpolation result by 0.0013.Experiments have verified the feasibility of the two algorithms in video multi-frame interpolation.Compared with Super-Slomo,the two algorithms increase the PSNR of the video interpolation result by 1.05 db and 1.14 db respectively.Compared with BMBC,it is based on volume.The product LSTM algorithm increases the SSIM of the frame interpolation result by 0.0043.