Research On View Synthesis For Multi-View Video System

With the increasing concerns and reliance on communications, com-puter network and digital multimedia, multi-view video will be the im-portant developing direction in digital video area in the future. Compared with traditional two dimension video, multi-view video could guarantee a better stereoscopic and immersive impression which make it match the hu-man visual requirements more closely. Moreover, multi-view video can provide users more options to change their viewpoints as they like. Conse-quently, multi-view video brings in completely new sensory experience for people, however, its own huge amount of data also creates great challenges for existing multimedia information processing, network transmission and other related technologies. Therefore, seeking the solutions for generating the high quality virtual viewpoint with limited reference views has become the internal requirement of developing multi-view video technology.To fulfill the basic needs and achieve the ultimate goals of the multi-view video system, we conduct our research centering on the studying of generating virtual viewpoints with limited reference views and propose a number of suitable algorithms about content presentation and view recon-struction for the system. The main contributions of this thesis are summa-rized as follows.1) A high quality disparity estimation system based on an integrated matching cost initialization algorithm is proposed. During the cost initiali-zation step, three individual cost terms are utilized to construct the cost volume:Gradient-based Census Transform (GCT), Absolute Color Differ-ences (ACD), and Gabor Pattern Differences (GPD). These cost terms can describe the relationship between the reference image and the target image from various aspects such as edge information, textural properties, color features and directional attributes. The whole system provides adequate supports to obtain the more accurate disparity maps.2) A stereo matching algorithm based on the Guided Filter and the Confidence-Mask is proposed. To improve the computational efficiency in the aggregation step, the Guided Filter is adopted to smooth the disparity volume. The main advantage of the Guided Filter is that it is an edge-pre-serving filter that can be implemented in a very fast way due to the run time in the aggregation step is independent of the match window size. Moreover, in order to eliminate the matching ambiguities brought by the winner-takes-all method, an effective disparity refinement approach using Confi-dence-Mask is proposed to select and refine the less reliable pixels. Both quantitative and qualitative evaluation show that the proposed method is comparable to state-of-the-art local-based stereo matching algorithms.3) A novel temporal consistency enhancement algorithm based on adaptive temporal gradient filter is proposed to eliminate the undesirable flickering artifacts caused by temporal discontinuous disparity sequence. The proposed algorithm not only avoid the high complexity brings by cal-culating the global energy function or the optical flow, but also has the capability of integrating any state-of-the-art image-based disparity estima-tion technologies and extending them to the spatial-temporal domain. It has many unique advantages such as wide range of implementation, low complexity, and high feasibility, etc.4) A Global-background Based View Synthesis (GBVS) approach is proposed to eliminate the artifacts areas along object boundaries caused by discontinuous values of depth map and missing information of the refer-ence viewpoint. In the proposed approach, the inter-frame information is utilized to generate a global-background image, then this global-back-ground image is applied to complement the images which have been pro-jected to the virtual view, and finally inpainting technology is employed to repair the still exposed regions. Additionally, compared with common view interpolation methods which depend on two or more reference views, the presented representation only requires one original view. Therefore it re-duces the occupancy bandwidth of network and provides a wider range for view synthesis. In summary, our work makes a number of fruitful attempts and sig-nificant progresses on stereo matching, temporal smoothing for disparity sequence,3D warping and image repair, etc. Accordingly, it takes a bene-ficial exploration to improve the development of view synthesis technol-ogy for multi-view video system.