The Data Fusion And Synthesis Algorithm For Multi-view Video Signal

As an emerging multimedia technology, three dimension video (3DV) has trans-formed the way people live, work and play. In comparison with conventional2Dvideos,3D videos could provide depth perception and realistic scene information, andofer a high quality and immersive multimedia experience. Despite the advantagesover2D videos, the characteristic of huge amount of data and complex correlationbetween them, throws new challenge for its popularization and application.In recent years, many researchers devote to the relative technologies about3Dvideo system, including media display, media content creation, media representationand compression, and media transport and storage. Nowadays, the standardizationprocess of3D video coding has started, and explains multi-view video or multi-viewplus depth (MVD) as the representation format for the3D scene. This paper mainlyfocus on the encoding efciency for multi-view video and the virtual view synthesizerat the terminal side, presents the corresponding efective algorithms and validates abetter performance comparing to existing methods.To improve the compression efciency by exploiting the inter-view correlationtogether with the temporal inter-frame correlation, Joint Video Team (JVT) has devel-oped the Multi-view Video Coding (MVC), an amendment to H.264/AVC standard.However, the high encoding complexity and huge amount of data exchange makes itunsuitable for the cases where the encoder side is strictly constrained in computingperformance or provided energy. To alleviate the encoding complexity and maintainthe coding efciency, distributed multi-view video coding (DMVC) scheme has beenproposed based on the Wyner-Ziv (WZ) theorem to exploit the inter-view correlationsat the decoder side without requiring a joint encoding of the views. In a typical DMVCscheme, two kinds of side information are available at the decoder side: temporal side information and inter-view side information. An appropriate fusion method betweenthem has significant influence on generating better reconstructed results and improv-ing the rate distortion performance. This paper presents a new temporal and inter-viewside information fusion algorithm for DMVC. The proposed algorithm leverages con-ditional random field (CRF) to exploit the temporal consistency and intrinsic geometryregularity in multi-view video sequences. Specifically, the side information fusion ofWyner-Ziv (WZ) frames is modeled by CRF with the temporal, inter-view side infor-mation and corresponding estimated error maps. The association potential models thelocal accuracy of the side information, and the interaction potential of the CRF modelrepresents the local geometry regularity. The parameters in CRF model are trainedfrom previously decoded WZ frame, and inference is performed on trained weights bybelief propagation (BP) algorithm to generate fused side information. The modelingaccuracy is validated through experimental results, which show a significant perfor-mance gain over the existing fusion algorithms.In3D video system, the quality of synthesized view has significant impact onuser experience. Virtual view synthesis is dedicated to generate arbitrary virtual viewsof the same scene from given inputs. In3DTV system, view synthesis technologiescan be classified into two categories according to whether corresponding depth infor-mation is utilized: depth image-based rendering (DIBR) and warping-based method.Although depth-based approaches allow for an efciency and accurate synthesis ofvirtual view, the dense depth estimation cannot yet be solved in an automatic manner.As an alternative to DIBR, warping-based methods deform the image content directlythrough a warping function which is based on the feature correspondence between theinput and target view. However, existing warp-based algorithms sufer visual distor-tions in which the disparity changes rapidly, and ghost artifacts for object boundary.This paper proposes a motion aware shape preserving warping-based scheme for viewsynthesis, allowing for graceful tradeofs between temporal coherence in the back-ground and shape preservation for the moving objects. Correspondingly, the motionsaliency map and global motion is extracted to encourage the warp of current frameto be the same as the previous one, except for the region with moving object. Fur-thermore, the salient curves are detected between reference frames to formulate thedisparity term, which enforces curves warped in the same location to avoid ghost ef- fect. Extensive experiments are validated by the visual comparison and demonstratethat the proposed scheme outperforms state-of-the-art warping-based methods.