A Virtual View Synthesis Algorithm Based On Image Inpainting

A3D blockbuster "Avatar" opened the prelude of "3D Year One", also made theconcept of "3D" burgeoned in the domestic market. The major televisionmanufacturers at home and abroad have been pushing for its3DTV. Benefited fromthe development of LSIC and information technologies,3DTV enter into commoners’homes. While the hardware manufacturers pushing the popularity of3DTV, the factthat3D content is extremely scarce has also been caught the attention by industryexperts and ordinary people. Once the phenomenon "buy back, but can’t use" occurs,it will cause huge impediment to the development of3DTV. According to incompletestatistics of domestic content market, the total of3D content in the film and televisiondoes not exceed1%, even the new produced content, the3D format is difficult toupgrade to more than10%in the short term.So, if you want3D technology integrated into daily lives of ordinary people,3Dcontent must be easily generated and quickly produced. It has been caught attentionby many television manufacturers and program companies, they are making greatefforts to solve this problem. Due to the imperfect technology of3D filming, high costof shooting and lack of new sources,2D to3D conversion technology has beenrecognized the best solution by industry. The virtual view synthesis is the keytechnology of2D to3D conversion. Depth Image Based Rendering is one ofnewly-emerging virtual view synthesis technology. DIBR uses reference image anddepth image to synthetize virtual view through3D image warp. It has the advantageof high drawing speed and small bandwidth requirements. The main problem of DIBRis the hole in virtual view. The hole generates when some invisible pixels in referenceimage become visible in virtual view. In order to get high quality of virtual view, wecombine DIBR and image inpainting to fill the hole. There are three ways to fill thehole: preprocessing to depth image, using multiple reference images, image inpaintingpost-processing. But preprocessing to depth image will cause geometric distortion andthe filling result is not so good when big distance between reference view and virtual view. Using multiple reference images needs big computation complexity and highbandwidth requirement. Simply using backgrounds, textures, the adjacent pixels to fillthe empty, or linear interpolation to hollow edge have not been very effective.In order to generate high-quality virtual viewpoint image, this study proposes avirtual view synthesis algorithm based on image inpainting. This method can bedivided into two parts, first we propose a virtual view synthesis algorithm based onimage inpainting, then improve the above algorithm, propose another virtual viewsynthesis algorithm based on depth image layering. This method can easily fill thehole and reduce the image distortion.The first part, we propose a virtual view synthesis algorithm based on imageinpainting post-processing according to the principle of DIBR and image inpainting.This algorithm combines preprocessing to depth image and image inpainting to fill thehole after3D image warp. Compared to conventional DIBR algorithm, this algorithmcan fill big hole and improve the quality of rendering image. It has achieved a certaineffectThe second part, the first algorithm is aiming at filling the hole after3D imagewarp, including preprocessing to depth and image inpainting. Due to big computationcomplexity and unsatisfactory filling result, we propose another algorithm. First wesegment the depth map to distinguish the foreground and background, then do imageinpainting to the edge of foreground which can easily generate hole. This algorithmreduces the computation complexity and the synthesis result is better.In order to achieve the goal above, we first distinguish the foreground andbackground by automatic segmentation to depth image. Then get the foreground ofvirtual view by DIBR to the foreground of depth image and foreground of referenceimage. Then mark the hole area according to parallex formula, after that inpainting thehole area to get background image without any hole. Finally, do image fusion to theforeground and background of virtual view to get virtual view image with highquality.