Research On Super-Resolution And Fusion Of Depth Map Based On ToF Camera

3D technology has broad application prospects and business value in the field of telemedicine, military training, education, games, entertainment,3D television and so on. The obtaining ac-curate depth is the key technology in these3D applications and3D researches. In recent years, with the attractive advance of3D technology, many applications, such as:human-computer in-teraction,3DTV,3D reconstruction, gesture recognition, augmented reality demand a higher requirement for scene’s depth information. These applications need more dense and accurate depth information, and need a faster acquirement for depth information. The laboratory which the author of this paper works in proposed a relatively complete innovative natural3DTV sys-tem, which aims to restore the true depth of the real shooting scene, and to provide a comfortable and credible3D audience viewing experience. The key technology of Natural3DTV system is to reliably acquire the real scene’s depth information, and use the depth information to recon-struct a plurality of viewpoints, then to watch the real scene with a free viewpoint system. Thus this viewpoint reconstruction technology demand a higher requirement for the accuracy of depth information.There are two major kinds of depth information acquirement methods in the current:pas-sive stereo matching and active depth acquirement. Passive stereo matching methods appeared earlier, the core technology of Passive stereo is stereo matching algorithms, the main idea is to capture a set of images of the same scene based on multi-cameras, then search for the matched point in the scene and the corresponding points in this group of images. But passive stereo algo-rithm can not obtain the depth information of occlusion area. And both local and global passive stereo algorithms have insurmountable flaws. Our lab was using passive stereo method to obtain the depth information of scene, and to achieve free viewpoint display based on these depth in-formation. But the depth obtained by passive stereo could not fulfil the requirement of free view display system, the3D impression need to be improved. Therefore, this paper uses active depth acquirement to improve the accuracy of the obtained depth information. Active depth acquire-ment technology uses active depth cameras to directly measure the depth of the real scene. One kind of these depth cameras Which is widely used is ToF camera (full name is time-of-flight camera), it can capture a scene’s depth information in real time, its working principle is to emit controllable frequency modulation light and measure the flight time of the modulated light to determine the depth of a scene. This paper does a series of experiments and researches Based on the principle of ToF depth camera,such as correcting the depth error which is captured by ToF camera, depth map super-resolution, as well as the depth integration which combined with the passive stereo matching algorithm. The major innovations and contributions of this paper are described as follows:1.This paper studies the optical imaging system of ToF depth camera. In order to improve the depth capture performance of ToF camera, this paper analyses the properties of two major kinds of ToF camera noise:environment noise and multi path noise, this work is based on a series of different types real shooting scenes and experimental results. and the models of these two kinds of noises are established. Then two innovative filtering algorithms are proposed for ToF depth map based on the noise models, these algorithms effectively eliminated these two kinds of noise of ToF depth map.2.Due to the principle of ToF depth camera, the resolution of the depth map acquired by ToF depth camera is limited. the depth map does not match with the high-definition color image. So ToF depth map must be enhanced for higher resolution. Existing image super-resolution algorithms can not well fit the characters of ToF depth map. Therefore, this paper presents a joint learning based ToF depth map super-resolution recovery algorithm. This algorithm lines with the characters of ToF depth map and uses the theory of compressed sensing. This algorithm built two over-complete dictionaries based on training depth sets to obtain high resolution ToF depth map with higher accuracy. This algorithm also contains a classifying and indexing system to boost the learning and recovering process.3In order to obtain the high accurate final dense depth map, this paper combines both the active stereo and passive stereo methods to obtain high accurate depth map. This paper designs the integration algorithm based on the ToF depth map, combines with the advantages of passive stereo matching algorithm. This integration algorithm designs a energe function of ToF depth map to guide passive stereo matching, and combines these two kinds of depth maps with a reliability cost function to fuse these two kinds of depth maps. The algorithm effectively eliminates part of the error of ToF depth map, and also improved the passive stereo matching errors in the repetitive areas and textureless regions. The experimental results showed that the integrated depth maps have higher accuracy than the ToF depth map and passive stereo matching depth map.