Research And Application Of Augmented Reality Technology Based On Stereo Vision

Augmented reality(AR)has become a hotspot in the field of computer science because of its characteristics of the fusion of virtual space and real space,which can enhance people's perception and interaction with the real world.Due to the anatomical complexity of the human body and the difficulty of lesion localization in a minimally invasive surgery based on endoscopic vision,AR technology has been applied to surgical navigation.It can provide a perspective navigation for clinicians by overlaying prior information such as preoperative images on surgical video frames in real time,so as to realize an intuitive three-dimensional visualization and an accurate localization of the lesions.This may improve the successful rate of surgeries.This paper focuses on exploring and researching the problems and solutions in the application of AR based on stereo vision in pelvic minimally invasive surgery.The main contents of this paper are as follows:First of all,3D/2D registration based on color consistency is proposed for the difficult registration of a 3D model reconstructed from a preoperative image and a 2D target image in an endoscopic view.The registration is achieved by using the partial exposed pelvic bone,which can shorten the time of manual registration and improve the accuracy of registration.The method of visible points detection of the 3D model and the construction of a simulated test environment are also proposed.In addition,camera localization methods based on 3D model and stereo vision are proposed to solve the problem of endoscope localization.The camera localization algorithm based on 3D model minimizes the photometric error of visible points of the model to optimize the camera pose under the condition of the 3D model aligned with the 2D target area.Experimental results show the estimated trajectory of camera localization has a high accuracy.However,the robustness of the algorithm relies on the following three conditions.Firstly,the shape of the 3D model is exactly the same as the shape of real object.Secondly,the 3D model is perfectly aligned with the real object.Lastly,most parts of the surface of real object can be observed.This camera localization algorithm has certain limitations in realistic surgical scenarios.Therefore,a camera localization algorithm based on stereo vision is further proposed in this paper.ORB(Oriented Brief),which has excellent characteristics of features,was adopted in feature extraction and matching for localization,and improved the stereo matching.Then it estimated the camera pose by using stereo bundle adjustment and stereo 3D point cloud optimization,and reduced the cumulative error by a local mapping optimization.Experimental results show the localization algorithm based on stereo vision has high positioning accuracy and satisfies certain real-time performance.Finally,the AR is applied to the simulation system of pelvic minimally invasive surgery navigation.The overall flow chart and implementation details of the system were introduced.The AR navigation technology was implemented through the pose optimization thread,local mapping optimization thread and AR display thread.The algorithms were verified in the simulation of pelvis surgery.Meanwhile,the qualitative measurement of the accuracy of the estimated pose by the overlap accuracy of the 3D model and the real object is proposed for evaluation without the ground truth.Experimental results show this system has a good AR display effect.