Study And Realization Of Stereo Matching Based Depth Extraction And Virtual View Synthesis In Dynamic Scenarios

Virtual view synthesis is one of the main techniques of Virtual Reality, Free view-point Television and other computer vision applications. The performance of these applications highly depends on synthesis quality, but there are two problems obtained through virtual view synthesis technique. One is that the synthesis accuracy is not high enough in static scenarios, and the other is that the synthesized video is temporally inconsistent in dynamic scenarios. Local stereo matching and depth image based rendering are two key components of virtual view synthesis. The current local stereo matching methods can not achieve high accuracy in textureless, depth discontinuities and slanted plane areas, however low matching accuracy will lead to bad synthesis quality. Depth image based rendering algorithms are not robust enough, because they can not solve low synthesis accuracy and temporal inconsistency problem at the same time. Focused on the purpose of enhancing synthesis accuracy and temporal consistency, in this thesis, an intensive study is conducted including both local stereo matching and depth image based rendering.Based on the framework of local stereo matching which uses minimum spanning tree for aggregation, in this thesis, a high accuracy method is implemented by optimizing matching cost computation and postprocessing. Considering hamming code obtained by Census transform can describe the spatial structure information of a pixel, a joint matching cost which combines hamming distance, gray and gradient differences is formed, which increases depth accuracy in textureless areas. For postprocessing, firstly a disparity repairment strategy based on horizontal scanning is performed to improve accuracy near depth edges. Secondly, a disparity refinement step based on plane fitting is conducted to reduce noises in slanted planes. The depth image based rendering in this thesis refers to view synthesis reference software, and a robust temporal depth enhancement filter is presented. This filter consists of motion detection, depth reliability check and temporal depth enhancement procedures. Firstly, motion detection divides scenes into static and dynamic regions, and only static regions will be filtered. Secondly, left and right consistency check is applied to obtain depth reliability information. Finally, the depth reliability information will be included by temporal depth filter to smooth depth sequences and improve depth accuracy at the same time.The evaluation of stereo matching algorithms shows that the proposed method achieves 22.1% and 27.8% bad pixel percentage reduction using Middlebury standard dataset and self-captured sequences respectively. For virtual view synthesis algorithms, the proposed method attains 24.6% drop of temporal inconsistency and increases synthesis accuracy by 1.9% in Bookarrival dataset. Experimental results demonstrate that the proposed local stereo matching and depth image based rendering algorithms have achieved significant performance, so they can provide reliable technical support for synthesizing high quality virtual view images.