Research On Frame Rate Up-Conversion Algorithm

Frame rate up-conversion is a useful video post-processing technique to interpolate the intermediate frames and convert the low frame rate sequence to higher frame rate. In recent years, high definition digital televisions and various high performance multimedia systems have emerged. Improving the frame rate can provide better visual quality and enable the viewer to enjoy natural and fluent scenes. As to the low bit rate video applications on mobile devices, improving the frame rate can restore the frames which were skipped in the encoder end due to limited bandwidth. This will effectively reduce unpleasant fatigues caused by original low frame rate video materials.Frame rate up-conversion algorithm mainly consists of two processes: motion estimation and motion compensation. Most FRUC algorithms nowadays are based on Block Matching Algorithm (BMA), which is computational efficient and easy-to-implement. However, the BMA always try to select the motion vector yielding the least compensation error and thus easily fail in plain and texture area. Moreover, the BMA pre-assume that all the pixels in one block must have the same motion trajectories. In real scenes, the objects don't really conform to the block boundaries. So a single block may contain several moving objects, which could obviously cause severe ghost effects. The motion compensation based on BMA is divided into two categories: motion trajectory interpolation and dual motion compensation. The trajectory interpolation is usually more accurate than the dual method, but it confronts with overlap and holes problems. Improper methods will produce blocking artifacts in overlap regions and blurring effects in holes. We proposed an effective motion estimation method based on motion segmentation and refinement of boundaries. This method first adopts an advanced motion smoothness constrained criterion to reduce the estimation errors in plain and texture area. Then we segmented the motion field to locate the motion boundaries and later apply variable size BMA on these regions. At last, we decompose and smooth out the motion field for non-boundary area. Experimental results show that our method can highly improve the field density and reduce block and blurring artifacts.As to the overlap and holes, we proposed a motion compensation method based on occlusion detection and edge-preserved interpolation. We first analyze the geometry and error distribution of motion field and detect out the occlusion regions. Then we apply overlapped block compensation to solve the overlaps and carefully handle the occlusions. In the end, we adopt an edge-directional method used in Super Resolution to interpolate the holes in our FRUC application. Experiments have shown that our method can solve the overlap and holes problem quite well and preserve most of the edges.As the last part, we researched on the application of our algorithm on 2D/3D videos. We combine the above methods directly to form the 2D video FRUC algorithm. The conducted experiments have shown the effectiveness of our 2D-FRUC method compared to other conventional methods. In the following content, we research on the depth-image based 3D video and proposed a low-complexity 3D-FRUC framework. The single view and multi-view experiments show that our 3D-FRUC method is very efficient compared to those methods without depth information adoption.