Research And Applications Of FPGA-based Real-time Depth Estimation In 3D Video System

With the development of computer vision, video acquisition technology and three-dimensional(3D) display technology, especially the autostereoscopic display technology, the key technologies of 3D video system has been widely studied. Developing 3D video has become the next generation of video technology. 3D video contains multiple views video and the third dimension information, depth information, which brings people strong visual perspective experience and immersive effect. Because the synthesized multi views video is provided for autostereoscopic display in multiview video system, the reduction of definition is inevitable. The high definition is more important for 3D video system than 2D video. In 3D video system, depth estimation is the method to generate the third dimension information, which affect the quality of display by depth based rendering and which is an obstacle of real-time performance of multiview 3D system. Definitely, the depth estimation is one key technology of 3D video system. 3D video system includes video acquisition, pre-processing(image rectification, depth estimation etc.), coding, transfer, display etc. Depth estimation located in stage of preprocessing releases the burden of processing in the receive end. The introduction of depth map in the 3D video system reduces the amount of data comparing to sending multiview video directly. However, depth estimation algorithms are complex. Its implementation is high time consumption and computation increasing with high resolution processing. Results of depth estimation have accuracy problems. So depth estimation is an key problem in 3D video system. Efficient real-time depth estimation has become a hot field of research in signal processing. This paper makes a deep research on some key technologies of depth estimation and stereo video system, the main contributions and innovations include the following aspects:1. Depth estimation with fusion algorithm optimizationDepth estimation algorithms are complexand depth map accuracy problems still exist, such as the object edge error, errors in low texture area, and errors introduced by the image noises. Generally a single matching algorithm performs well in specified area of image and resutes in errors in other area. A fusion algorithm strategy is proposed. Census Transform enhances the edge preserving and SAD enhances the area with rich textures. The aggregate with crossing region and post-processing methods are used. The algorithm adopts hybrid matching algorithms can effectively reduce the errors caused by a single method. The ability of parallel processing and hardware implementation also are concerned.2. Edge preserving interpolation method based on depth featureConsidering the characteristics of depth map which contains large smooth areas and sharp edges and the growing application requirements of high definition(HD), An up-sampling method based on adaptive bilateral filter is proposed to realize the high definition, high quality depth map interpolation. The information of high resolution texture image is involved in the up-sampleing processing. Up-sampling processing improves the quality of the depth map by using of bilateral filter based intoplation, especially the object edges are preserved,that is benefit for high quality rendering processing. This method provides a reference solution for high resolution data processing in real time system.3. FPGA-based high resolution real-time depth estimation systemConsidering the requirements of real-time processing and high resolution television application, a solution of depth estimatin and an implementation of full High Definition real-time depth estimation system are proposed and completed. The system is implemented based on fashion FPGA device and high efficiency hardware language and synthesis software. The implementation utilizes efficient parallel architectures, fine pipelines, asynchronous FIFO and clock domain crossing to realize real-time processing of complex depth estimation algorithms. Depth estimation system achieves the processing speed of 125 fps, the 240 pixel disparity search range, full HD(full High Definition, 1920 × 1080) image resolution. The proposed depth estimation system can be used in real-time 3D television system as a depth estimation engine. And the core module of depth estimation can be used in embedded system design or ASIC transplantation as an independent hardware module, or be involved in the integration design of low power applications.4. The real scene HD 3D reconstruction video system based-on depth estimation and other applicationsOne of main problem of real-time stereo system is the performance of key techniques in different processing stage. Considering real-time requirments of stereo video system, we proposed implementation based on depth estimation based on FPGA hardware, fast coding, virtual view rendering based on parallel, etc. The high resolution, fast stereo video system with autostereoscopic display is implemented. The frame rate can reach 22-25 frames per second. The real-time depth estimation module can be utilized in embeded design besides the stereo video system application.