Research On Real-Virtual Merging And Occlusion Handling In Stereoscopic Augmented Reality

Graphical based predictive control is an effective means to overcome the problem of large time delay in teleoperated robots. And application of binocular stereoscopic augmented reality to graphical based predictive control of teleoperated robots can play an important role in improving the efficiency and accuracy of the system. Several key technologies of binocular stereoscopic augmented reality application to graphical based predictive control of teleoperated robots are studied in this paper.First, capturing, merging and synchronizing of binocular stereoscopic video stream and dynamic integration approach which mainly includes the image registration algorithm within binocular stereoscopic video is studied.Second, a novel approach for multi-layer real-virtual occlusion handling in real-time augmented reality applications with two cameras is proposed, which combines 3D reconstruction based on stereo correspondence and 3D pose estimation based on feature tracking. Two-threaded structure and key frame based approach is used. Information for 3D reconstruction is extracted from pairs of key frames, meanwhile a 3D tracking process is running through inter frames. Models built for real objects are kept in use during the tracking process until new ones are finished.Third, the approach of multidimensional human-computer information interaction in augmented reality scenes of teleoperation is studied. Omnibearing information feedback of remote site is provided to the operator through multiple channels such as visual images, text, sound, haptic feedback, etc. Perceptible and imperceptible information is integrated and presented to the operator intuitively, which provides powerful assistance to fulfill the teleoperation task.Finally, a teleoperation test platform based on binocular stereoscopic augmented reality is built, on which relevant theories and algorithms are implemented and validated. Experimental results show that the integration of binocular stereoscopic augmented reality and graphical based predictive control can improve the operational effeciency and accuracy of teleoperated robot systems.