Research On Depth Based Multiview Video Coding And Image Enhancement

With the development of multimedia technologies,3D video has developed rapidly in recent five years. Among many representation of3D video, Free-view video(FVV) which allows users to select the views and angles has become a popular research field. In2007, MPEG proposed multiview plus depth(MVD) as the format of FVV. It redues the number of views needed to be transmitted and the arbitrary virtual views are generated by depth image based renderding algorithm. Compared to2D video, MVD includes more views with multi-texture views and multi-depth views. The introduction of depth information is exciting and provides both opportunities and challenges for MVD coding and image processing.Firstly, the acquisition of depth information that describes the3D structure of scene is the premise. Depth captured from Kinect are become as the trend in recent years. But there are depth missing areas in the depth image due to the occlusions and other factors, which influences the quality of depth image and virtual views. Therefore, it is crucial to research the efficient depth map holes filling algorithms for high quality depth acquisition.Secondly, the incresment views number of MVD suffering from the radically rising of data volume. So, the effective compression of MVD is urgent. On one hand, because the depth map is not used for display and its characteristics are different from color image, the compression of depth image by traditional video coding technologies will cuase the lose of depth edge information, which reduces the quality of virtual views; On the other hand, In traditional multi-view video coding, the translational motion model used in disparity compensated prediction is not sufficient enough to remove the interview redundancies because the non-translational motion may exist between different views. Therefore, researching the effective MVD video coding technologies based on MVD characteristics is essential to promote coding efficiency and quality of virtual views.Thirdly, in the application of multiview surveillance, it is essential to enhance the image captured under low light environment due to low contrast and high noises. The traditional methods mainly uses the information of2D image during the enhancement, which suffering from losing of the depth perception in the enhanced image. The depth perception is important for human during the image perceiving. The depth image and color iamge are the differnet descriptions of the same scene and the former one is less affected by low light environment, which provides the robust reference information for low light iamge enhancement. So, researching to utilize the characteristics of depth iamge to enhance the quality and depth perception of low light depth iamge is crucial. Faced with above requirements and challenges, we researching on the depth information based multimedia video coding and image enhancement. In this thesis, the depth holes filling, depth views video coding, texture views video coding and depth based low light image enhancement technologies are researched. The main contribution and innovation are follows:(1) Similarity Constraint Sparse Reprentation for Depth Holes FillingIn order to avoid the influence of uncorrelated non-holes pixels, we propose similarity constraint sparse reprentation for depth holes filling algorithm to calculate the optimal weights by solving a constrained least squares problem based on collocated patches in color image. We only chooses the most relevant patches adaptively instead of all the patches to represent central patch, leading to reconstruct sharp depth contours at discontinuous regions. Experiments results demonstrate that the Kinect depth maps processed by our algorithm achieve more accurate depth contour and better quality in local detail region than some state-of-art methods. The PSNR gains is1.8dB.(2) Structure Tensor based Median In-Loop Filter for Depth Video CodingIn order to improve depth video coding efficiency and virtual views quality, we propose a structure tensor based median in-loop filter for depth video coding. By finding the reliable neighbor region using4-D structure tensor analysis, the neighbor range is narrowed down to the region in the same depth plane with the current depth pixel. Then, instead of calculating the weighted mean, median operation is performed among the candidate depth pixels in reliable neighbor region to select one depth pixel as output, which not only can avoid the influence of outliers but also doesn’t introduce as many new depth values. Compared with anchor methods, the bitrate of our depth video coding method can be redued6%and the quality of rendered views are better.(3) Adaptive Learning based View Synthesis Prediction for Multiview Video CodingIn order prevent the existing VSP techniques from signal mismatches caused by depth distortion and illumination mismatches, we propose an adaptive learning based view synthesis prediction algorithm which enhance the predication capability of virtual view picture by adaptively adjusting the view synthesis process according to previous decoded information. Experiments show that the quality of the virtual view picture has been improved significantly up to3.42dB. The bitrates reduction could be achieved about11%compare to existing VSP techniques.(4) Depth perception based low light image enhancementThe existing low light image enhancement algorithms merely use the2D plane information such as color cues contained in an image to enhance the low light image. The depth information of a scene is not considered, which results in loss of depth perception in the enhanced image.In this thesis, a depth based method for low light surveillance image enhancement is proposed. Pre-processing for Kinect depth map, depth constrained non-local means denoising and depth-aware contrast stretching are performed successively in our algorithm to promote the visual quality for low light surveillance image. Comparing with the previous works, our method is able to enlarge the low dynamic range and promote both globe and local depth perception for the low light surveillance image meanwhile. The experimental results show that our method generates clearer object edges and more distinct depth perception for enhanced low light surveillance images. The PSNR gain of denoised image by our method is0.4dB.In summary, by analyizing the characteristics of multiview and depth views in MVD, we have researched the depth based multiview video coding and image enhancement. In this thesis, the proposed four algorithms have further exploited the application potentialities of depth information and impoved the mulitiview plus depth video coding efficiency.