Research On Real-time Virtual View Synthesis System Based On Binocular Camera

Three-dimensional display technique has gradually become a key technique in TV display systems and human-computer interaction systems.It has been used in many applications with broad commercial prospects,such as 3D-TV,3D games,VR,AR and so on.The naked-eye 3D display technique frees users from wearing equipment,provides free-point views and thus becomes a pop research.But nakedeye 3D content generation is complex and costly because it needs many images from different viewpoints at the same time.By developing a real-time virtual view synthesis system,the system can synthesize virtual views with low cost and high efficiency based on two reference views provided by binocular camera,and realize real-time generation and display of naked-eye 3D content.The disparity information is firstly recovered from corrected binocular images through stereo match algorithm.To balance the speed and accuracy of the algorithm,semi-global stereo match algorithm is taken.To solve the holes and noises in the recovered disparities,a combined disparity refinement scheme based on consistency checking,filtering and disparity repainting is proposed.It can fill holes and filter noises,thus can effectively improves the estimation accuracy.Based on the disparity estimation of stereo match algorithm,depth-image-based rendering(DIBR)technique is improved to synthesize virtual views.Firstly,rendering with single reference view is extended to double reference view,which can reduce the production of holes.Secondly,a self-adaptive image repairing scheme based on background extrapolation and mirroring is designed,so the advantages of both is combined.Finally,a depth gradient eroding process is proposed to remove the common fake artifacts in DIBR technique,which can effectively reduce the image quality degradation caused by fake artifacts.Considering the fact that there is a great content correlation between the former and the later frames of continuous scanning video,the background table is proposed to extract and store spatiotemporal background information,which makes up for the lack of reference information in the rendering process.In the circumstance of static camera,the background table is fused into the image rendering process,including background table query,image supplement and background table update.In the circumstance of moving camera,SLAM technique is applied to estimate the camera pose in real time,and the background table is transformed according to the estimation.The use of background table can effectively correct distortions produced in image rendering process and realize the accurate rendering of virtual views.To meet the need of real-time stereo-based view synthesis system,the hardware platform of the system is built,and the software system is designed and implemented.The system consists of the base modules and the interactive module.The base modules run in parallel with stereo matching module,image rendering module and SLAM module.The interactive module completes the synthesis of stereo image and the control of the system.And the real-time processing speed is achieved using GPU parallel acceleration.The overall performance and the subsystems performance have been tested by a large number of experiments,and the improvement scheme of the system has also been verified.