Research And Implementation On Real-time Three-dimensional Content Generation Algorithms

Three-dimensinal display technology can provide people with realistic visual feelings, so it has been the hot topic of the display field. In recent years, many manufacturers have launched a variety of three-dimensional (3D) displays, which can be classified into binocular stereoscopic displays and autostereoscopic displays from a technical perspective. For the same 3D display, the quality of 3D contents determines whether users can achieve immersive 3D feelings. Since the 3D display is a new technology, traditional 2D contents can't be viewed on 3D displays and different 3D displays require different 3D contents. For example, binocular stereoscopic displays support binocular stereoscopic contents and autostereoscopic displays support multi-view contents. With the development of 3D displays, users want to see more abundant 3D contents. However, the lack of 3D contents has become a issue because of expensive equipment costs and the limitation of 3D production technology.A effective way to solve the issue of lacking of 3D contents is producing 3D contents from existing medias by view rendering algorithm. Based on the existing 3D display system platform, this paper focuses on the realtime 3D content generation algorithm, which can be classified into the 3D content generation algorithm based on dense depth map and the stereo to multi-view conversion algorithm based on image warping. The achievements are listed as follows:1. To achieve the realtime performance, a realtime 2D to 3D conversion algorithm based on dense depth map is proposed. The proposed algorithm takes advantage of the depth model, the warm and cool colors, and the image saliency to generate the dense depth map of the reference view. Then the backward depth-image based rendering (DIBR) technology is used to render virtual view images without blank regions. Finally, the reference view and virtual view images are spliced to generate the stereoscopic images, which can be watched on binocular stereoscopic displays. In most cases, the proposed algorithm can generate temporal stable dense depth map with high quality, and present good stereo effect. By experiments, the proposed algorithm can render full HD videos in 34.4fps with Graphic Processing Unit (GPU) implementation. Meanwhile, this paper has implemented the proposed algorithm on Field-Programmable GateArray (FPGA) architecture. By experiments, the FPGA implementation can process full HD videos in 30fps.2. To achieve the realtime performance, a realtime stereo to multi-view conversion algorithm based on dense depth map is proposed. The proposed algorithm combines the idea of variable support window with the architecture of dynamic programming algorithm to generate the dense depth map of input stereoscopic images. Then the Shift-Sensor camera model is used to design the parameter of virtual camera array and DIBR technology is used to generate virtual view images with blank regions. For the virtual view image which is between the reference views, the complementary of disoccluded areas between reference views is used to fill the hole regions. Finally, the remaining blank regions are filled with simple interpolation algorithms. When the camera parameters of reference view are known and the scene is colorful and richly-textured, the proposed algorithm can generate high quality depth map, and render high quality virtual view images between reference views. By experiments, the proposed algorithm can render 8 virtual view videos with the resolution of 640×360 in 20.9fps with GPU implementation. Meanwhile, this paper has implemented the virtual view rendering module of the proposed algorithm on FPGA architecture. By experiments, the FPGA implementation can output full HD autostereoscopic videos in 30fps.3. A realtime stereo to multi-view conversion algorithm based on image warping is proposed, which is inspired by the virtual view rendering algorithm based on image warping. This paper improves the perfomance of the algorithm with two main contributions. As a first contribution, adaptive meshing is introduced into the image warping approach, which aims to reduce the computational complexity on the premise of good quality for virtual views. As a second contribution, sparse stereo correspondence algorithm is used to generate the sparse disparity map, which is inspired by the local stereo correspondence algorithm. Compared to the feature extraction algorithm, the proposed algorithm is able to get more uniform distributed sparse disparities with high matching precision in realtime. By experiments, the proposed algorithm can render 8 virtual view videos with the resolution of 1280×720 in 21.3fps with GPU implementation.