A Real-Time High-Quality System For Depth Image-Based Rendering

Extending visual sensation to 3D vision has been studied for decades and it is widely used in many applications,such as three dimensional television(3D-TV)and virtual reality gaming.Many methods are proposed to provide the audience with a greater sense of presence in a computer-generated environment,and depth image-based rendering(DIBR)techniques have drawn more attention nowadays.DIBR,which efficiently generates novel realistic viewpoints using the known original views and the depth information,has been treated as a key part of 3D display systems.In common DIBR algorithms,the production of high-quality virtual views not only poses challenges to the accuracy of the depth maps,but also requires additional inpainting steps to fill in the occluded regions that become visible in the synthesized views.It is noted that DIBR is a complicated and time-consuming procedure,which makes it hard to process in real time on a CPU,especially for high resolution images.Current research efforts on DIBR focus on the high-quality performance and real-time processing speed.This thesis proposes a high-resolution high-quality DIBR hardware which consists of image rectification,disparity estimation and view synthesis.The proposed hardware is verified in real-time with a complete system from the initial image capture to the virtual image store.Image rectification is an important pre-processing step of the DIBR system to solve the cam-era misalignments.This thesis proposes a novel TPCL image rectification algorithm and its real-time hardware implementation.The first phase,which determines the transformation matrices for the original images,is calculated only once by software when the system starts.The second phase,which utilizes the transformation matrices to generate the rectified images,is implemented on an FPGA with computational logic.It achieves a processing speed of 50 frames per second(fps)for 1080p resolution.Evaluated on images that are captured from the real world,it can provide satisfactory rectification results.Disparity estimation takes a pair of rectified images,estimates the displacement of each pix-el between the two images,and displays the associated movement in disparity maps.This thesis proposes a local EMCSASW disparity estimation algorithm that focuses on depth discontinuities and disparity smoothness.Considering the two factors,the proposed algorithm has two main con-tributions:1)it combines mini-census transform and segmentation-based adaptive support weight to get accurate disparity maps in depth discontinuities;2)it presents a segmentation-based refine-ment which consists of consistency check,disparity voting,disparity inpainting and median filter-ing to improve disparity smoothness.Moreover,a sophisticated hardware architecture is presented with hardware-oriented optimizations,efficient parallelism scheme and data reuse techniques to accelerate the proposed EMCSASW algorithm.It is designed to achieve a high throughput,and can be scaled up easily.The hardware implementation can process 45 fps at 1080p resolution for a 64 pixel disparity range.Evaluated on selected image sets of the Middlebury benchmark,the average error rate of the disparity maps is 6.02%.In addition,selected image sets of the KITTI benchmark and images captured from the real world are also used for quality evaluation.View synthesis utilizes rectified images and the corresponding disparity maps to generate virtual views.This thesis proposes a novel EVIE view synthesis algorithm that consists of view interpolation and view extrapolation.Different rendering and inpainting methods are used de-pending on the target location of the virtual viewpoints.The hardware architecture of the pro-posed view synthesis algorithm is designed,and row-based ping-pong buffers are used to provide pipeline processing.It is implemented on an Altera FPGA,and can generate 1080p virtual images at 65 fps.Evaluated on selected image sets of the Middlebury benchmark,the average peak signal to noise ratio(PSNR)and structural similarity(SSIM)values of the virtual images are 30.58 and 0.9423,respectively.The above three parts are integrated as a complete DIBR system.It is implemented on an Altera EP4SGX530 FPGA at a processing speed of 45 fps for 1080p resolution.Selected image sets of the Middlebury benchmark and images captured from the real world are evaluated to prove the high quality of the DIBR system.The average PSNR and SSIM values of the virtual images from the Middlebury benchmark are 30.07 and 0.9303,respectively.