Mixed Reality 3D Registration Technology In Remote Assistance Of Surgery

In the course of mixed reality remote clinical treatment or surgery,the remote guidance end marks the surgical site on the virtual model of the 3D reconstruction of the patient's organs to complete the surgical guidance to the local end.The local doctor manually registers and displays the reconstructed organ model and the marked information in the actual surgical scene through the head-mounted lens.However,the remote guidance terminal only 3D reconstructed the patient's organs,which caused the remote guidance terminal to have a weak sense of true immersion in surgical assistance,and the orientation of the marked information of the guidance was not consistent with the actual surgical scene.In addition,the local end uses manual registration for the registration of the virtual model,so the operation of the wearer of the mixed reality device greatly affects the registration effect,so that the registration accuracy cannot be effectively guaranteed.In view of the current problems in remote assistance of mixed reality surgery,this thesis combines a 3D reconstruction of the surgical scene at the remote guidance end and 3D registration of the local virtual model,and proposes a mixed reality registration technology for remote assistance of surgery.Specific research contents include:Firstly,in order to improve the accuracy of image matching in the remote assistance of local registration in surgery,an improvement is made to the ORB(Oriented FAST and Rotated BRIEF)image matching algorithm.First,the SURF(Speed Up Robust Feature)algorithm is used to detect feature points to obtain more uniformly distributed feature points.Then,before the RANSAC algorithm eliminates the mismatch,the Canny algorithm is used to remove the point pairs with a large difference in the number of edge points in the neighborhood of the feature points,so as to reduce the matching delay and improve the matching accuracy.Secondly,in order to enhance the true immersion of the experts in the remote guidance terminal of the surgical remote assistance to the surgical assistance guidance,a 3D surface reconstruction of the surgical scene was carried out.Using the method of 3D image reconstruction based on multi-view images,first use SFM(Structure From Motion)algorithm and PMVS(Patch based Multi-view Stereo)algorithm to reconstruct the dense point cloud of the surgical scene.Then the dense point cloud is analyzed,and the point cloud is enhanced and optimized based on the least square method to make up for the holes and errors generated during the reconstruction process.Finally,Poisson reconstruction and texture mapping are performed on the enhanced and optimized dense point cloud to obtain a 3D surface model of the surgical scene.Thirdly,on the basis of the above research,a mixed reality registration technology method for remote assistance for surgery is proposed.The remote guidance terminal sets the display position of the virtual surgical instrument model according to the reconstructed surgical scene model.The local end uses the sparse point cloud of the surgical scene as matching prior information,uses the real-time captured image of the surgical scene to find the corresponding key image in the point cloud,and realizes the registration of the virtual surgical instrument model according to the camera pose.