Research On Video Frame Interpolation Based On Implicit Motion Estimation

Frame interpolation is a key technology in the field of video enhancement.Its main task is to predict in-between frames by using adjacent frames.Frame interpolation plays a significant role in many applications,such as frame rate up-conversion,slow motion effects,video super-resolution,and video compression.The key point is how to ac-curately extract motion information.Explicit-corresponding based methods have been widely studied in the past,but finding corresponding is often sensitive to occlusion,lighting changes,and motion blur.In the last two decades,considerable progress has been made in this field.However,finding accurate correspondences usually needs it-erative global optimization algorithms which is computationlally expensive.Recently,a lot of progress has been made in implicit motion estimation based frame interpola-tion such as phase-based methods and convolutional neural network based methods.Implicit motion estimation extracts motion information by using the local conversion instead of by using global optimization.However,challenges under difficult conditions still remain.On one hand,conditions such as extreme lighting changes and large dis-placement are still difficult to handle;on the other hand,although frame interpolation algorithms based on convolutional neural networks have achieved improved results,they usually contain a large amount of network parameters,and only focus on single frame interplation.Based on the above research background,this thesis has made an in-depth study on frame interpolation algorithms based on implicit motion estimation,such as phase-based methods and convolutional neural network based methods,and handles the challenging problems by combining them.The main contributions of this thesis is as follows:To enlarge the range of motion that phase difference can encode,a frame interpo-lation algorithm based on phase correction and amplitude alignment is proposed.The algorithm uses top-down phase preserving and bottom-up phase correction to distin-guish objects in different motion ranges,and then uses the corrected phase difference information to align the corresponding amplitude signals in the intermediate frames.In order to further improve the image quality,post-processing algorithms are designed for different applications such as color images and depth images.The experimental results show that the proposed algorithm has obvious advantages in depth-map frame inter-polation.For dataset with larger motions,the PSNR score is increased by 0.1-0.2dB;the proposed method handles better under conditions like brightness changes than other method and has competitive performance on high frame rate sequences.In order to interpolate multiple in-between frames in one pass and handle chanl-lenging scenarios such as extreme lighting changes,a multi-frame interpolation algo-rithm based on convolutional neural network is proposed.The algorithm is based on an encoder-decoder network and uses a multi-stream structure to model the relation between optical flow fields of multiple in-between frames to enhance the temporal con-sistency.A phase sub-network is introduced to extract the phase features of the image,which is used to deal with extreme lighting changes.Finally,a synthetic sub-network is used to further improve the image quality.Experimental results show that the pro-posed method is superior to state-of-the-art methods in different testing datasets.For the dataset containing larger motions,the PSNR score is increased by 0.4 dB on average and it preforms well under extreme lighting changes.In order to reduce the parameter amount of the frame interpolation network,a lightweight frame interpolation algorithm based on a single decoder neural network is proposed.The algorithm uses one-dimensional separable Gabor filters to extract phase amplitude feature pyramid which can efficiently encode motion information.A sin-gle decoding network is then used to predict the optical flow field,followed by image warping and context-aware synthesis sub-network.The network is trained on quintets by using motion linear loss.Experimental results show that compared with state-of-the-art methods,the proposed method can reduce the network parameters by an order of magnitude while maintaining competitive performance.For the dataset containing large motions,the PSNR score is increased by 0.28 dB on average.