Research On Video Pre-Processing Algorithms For Natural 3DTV System

Compared with the conventional 2D television and some existing virtual 3D applications, the Natural 3DTV we proposed is dedicated to provide a 3D representation of the real worlds. By delivering stereo image pairs to the left and right eyes individually, Natural 3DTV is able to give the viewer an extraordinary feeling that they themselves are in the real scene. Because of its incomparable way to display video sequences, it is not hard to predict that Natural 3DTV would definitely play a leading role in the future's television development, and bringing brand new innovations that would change the whole chain of TV industry.Based on the existing 3D-Capturing and Displaying platform in our laboratory, this paper is focused on the key algorithms dealing with the 3D video pre-processing procedures, including: multiple camera calibration and their image geometric rectification.Different from the traditional 2D video capturing systems, 3DTV, which has to reconstruct the three dimensional information from 2D images, requires multiple cameras to be used to capture the dynamic scenes at the same time. Therefore, it is quite necessary to find an easy, practical and high performance way to calibrate multiple cameras. Here, after a comprehensive investigation of the multiple view geometry, we proposed a joint multi-camera calibration technique by rebuilding the original calibration model to construct a homography measurement matrix. In the calibration process, many state-of-art algorithms in 3D reconstruction are introduced to deal with the missing data in the measurement matrix, thus cameras are not required to observe the model plane placed at all orientations. This not only improves the robustness of the calibration result, but also makes the algorithm a convenient choice for practical use.On the other side, rectification of the images from different cameras is also necessary to ease the post image processing algorithms. Here, we first set the images from the main view to be the reference, and then rectify the images from the other views separately to create multiple stereo image pairs. Secondly, each image pairs are processed to compute the depth information and extract the occlusion areas. Finally, we put all information from different image pairs together to build the final estimations of the real scene.