Research On Key Technologies Of Depth-map Estimation For Free Viewpoint Video

People can perceive the depth while watching a stereoscopic screen; they can change adifferent viewpoint according to their own preferences while watching multi-view video. It isdifficult to achieve the effects by the ordinary two-dimensional video, and is more and morepopular with audiences. Therefore,3D and multi-view video is one of the mainstreamdevelopment directions of today’s video technologies and applications, and has a very broadmarket prospects. But there are a number of key technologies that need to be solved: multi-viewtexture and depth acquisition, multi-view coding, transmission and decoding, and multi-viewrendering. In particular, how to obtain accurate depth information has become one of the hotresearch areas. The focus of this dissertation is how to introduce interactivity to the stereoscopiccontent, obtain depth information of the corresponding scene in the multi-viewpoint videosystem, and how to use the depth information for multi-view rendering, so that the audience canfreely select the viewpoint position.By building a system of free viewpoint video, we carried out the research work of three keytechnologies: depth maps estimation from a single viewpoint motion video, robust disparitymaps (depth maps) estimation from multi-view video, and virtual viewpoint synthesis based onDIBR. The major work and innovation are as follows:It is very difficult to estimate depth map from a single viewpoint motion video. By usingSFM (Structure from Motion) and the optical flow calculation, the image frame in the singleviewpoint motion video is converted into textures+depth map format, and each frame has anassociated camera pose. Firstly the algorithm makes use of the SFM to obtain scene sparserepresentation, the matching relationship of the feature point between image frames, and thecamera pose of each key frame. After a reference frame is given, the algorithm selects anotherimage frame which is concerned with the motion flow calculating by using the qualitymeasurement, in order to obtain the optimum depth maps. The algorithm does not carry out anydepth map interpolation calculation.In order to obtain a reliable depth map in low computational complexity, in this thesis, theglobal optimization methods integrated into the framework of the regional matching. There aretwo different ideas of the algorithm in order to extract depth information and to get more reliabledisparity map from the stereo image pair: One is the global error method based on energyminimization, and another is based on the linear region growing. In the block matchingalgorithm, In the search range of parallax, the global error energy minimization methodconstructs an error energy for each of the parallax matrix by block matching techniques. Linear growth to determine whether the error energy of correlation degree between a certain points withthe root point is less than the threshold value set in advance in the growth process. Every errormatrix is repeatedly perform mean filtering to find the minimum error energy, and thus to obtaina disparity map and depth map. The algorithm uses the smoothing function to do globaloptimization. In order to improve the reliability of the disparity map, the algorithm eliminatesunreliable forecast point due to occlusion area, by detecting the error energy in the disparity map.In the matching process, regional matching algorithm has been optimized through the applicationof recursive mean filtering fast calculation method and consequently it improves the efficiencyof the algorithm.Aiming at the ambiguity of the binocular stereo problem that may be caused by low textureareas and occlusions, a fast stereo matching based on Daisy feature descriptor and modifiedweight kernel is proposed in this thesis. Firstly, the local Daisy feature descriptors areconstructed fast and densely for both stereo pairs, the initial matching costs being calculatedfrom the features; two-pass aggregation with modified weight kernels for the reliable matchingcosts are applied to resolve ambiguity of feature similarities；Initial disparity map is obtained viaWinner-Takes-All optimization from them. Secondly, in order to improve the quality of disparitymap, this thesis adopts sequentially the refining procedure with modified bilateral filtering,symmetric consistency check and multi-directional weighted disparity extrapolation. Theexperiments indicate that this technique with concise structure and low complexity can improveeffectively the matching accuracy and obtain comparably accurate and piecewise smooth densedisparity map.This thesis presents a DIBR virtual viewpoint rendering algorithm based on the depth ofinformation processing. The algorithm makes use of camera parameters and depth of information,the reference viewpoint images and depth images are wrapped to the virtual viewpoint. Cracksand artifacts are removed by post-processing the projected images based on the depth ofinformation. Using the depth information, the remaining disocclusions are filled by inpaintingtechniques, and then these images are blended together. The experiments indicate that themethod is superior to the existing methods of generating virtual, not only in the subjectivequality but also in the objective quality. The study compares the quality of the composite imageof the multi-viewpoint rendering algorithms with the old, and analyses the influence of the videocompression.