| Surgical navigation system is an intelligent device that combines medical image segmentation,computer vision,spatial localization and visualization technologies to assist in the localization of surgical targets in surgical procedures.The surgical navigation system needs to analyze the patient’s medical images,determine the surgical navigation target,and show the tumor location to the surgeon through visualization technology to improve the accuracy of the surgery.For surgical procedures such as tumor resection,it is important to improve the accuracy of surgical navigation systems as surgeons need to determine the location of the tumor and its boundaries intraoperatively in order to accurately remove the tumor and avoid recurrence due to residual tumor tissue.However,existing surgical navigation systems are inadequate in preoperative image segmentation,especially the accurate segmentation of multimodal medical images is still a challenge.In addition,existing intraoperative visualization techniques are mainly based on screen displays,leading to frequent switching of the surgeon’s view between navigation information and the surgical area,which distracts attention and affects hand-eye coordination.In the emerging projective augmented reality surgical navigation,precise surgical navigation error analysis and comparison with screen-based display navigation systems are difficult areas of current research.The current problems can be decomposed into the following three key issues:(1)revealing the intrinsic association of multi-scale unimodal-multimodal potential semantic information in multimodal medical images;(2)resolving the image-patient position alignment relationships in the surgical space,target space and guidance space;and(3)analyzing the errors of projective augmented reality precise surgical navigation.To address the above problems,this study proposes a precise surgical navigation system that realizes preoperative artificial intelligence multimodal medical image precise segmentation and intraoperative projected augmented reality stereotactic localization and visualization precise navigation.The main work of the study includes:(1)To address the first key problem,this study proposes modality preserving encoder(MPE)and modality fusion decoder(MFD)modules and constructs MPU-Net(Modality Preservation U-Net),a deep neural network for multimodal medical image segmentation.MPU-Net is based on multiple modality preserving encoders and modality fusion decoders,which combine MPU-Net achieves accurate segmentation of multimodal medical images by fusing multi-scale and multi-level unimodal and multimodal information.Experiments show that MPU-Net achieves the first and second place in the segmentation of brain tumor and prostate,respectively.The ablation experiments show that the design of MPU-Net has significant advantages.(2)To address the second key issue,this study proposes the coaxial projectionbased augmented reality stereotactic positioning and visualization system sCPI(stereotactic Co-axial Projection Imaging).sCPI system simplifies the camera-projector calibration process in stereotactic positioning by using a beam splitter to configure the monocular camera and projector on a common optical axis.and enables intraoperative visualization of the navigation information projected directly in the surgical area.The minimum projection resolution and camera-projector coaxiality of the sCPI were evaluated,and the tangential position error and positioning error distribution characteristics of the sCPI stereotactic positioning and visualization system were investigated experimentally.The experiments show that sCPI has high stereotactic positioning accuracy and also visualizes the information of intraoperative navigation by in-situ projection.(3)To address the third key issue,IsCPI(Intelligent Stereotactic Co-axial Projection Imaging),a precise surgical navigation system based on MPU-Net and sCPI,is proposed in this study.IsCPI can perform intelligent segmentation of preoperative multimodal medical images and use the segmentation results as surgical targets.IsCPI enables stereotactic and in situ projection surgical navigation through image-patient alignment,parallax correction,and real-time patient tracking.In this paper,the preoperative image segmentation error and its distribution,and the intraoperative target projection localization accuracy of IsCPI are investigated with brain tumor as an example.The localization accuracy of IsCPI and the conventional navigation stick and monitor-based surgical navigation system were compared horizontally through phantom experiments,which is the first time to compare the augmented reality stereotactic surgical navigation system and the conventional surgical navigation system horizontally through phantom experiments.In this paper,the overall navigation error of the IsCPI precise surgical navigation system is quantified and the error components are analyzed through brain tumor phantom surgical navigation experiments.Finally,the feasibility of the proposed surgical navigation method for clinical use is verified through clinical demonstration experiments.The significance of this study is to propose a precise surgical navigation system that achieves preoperative artificial intelligence multimodal medical image precise segmentation and intraoperative projective augmented reality stereotactic positioning and visualization precise navigation.This study can improve the accuracy of the surgical navigation system and promote the development of medical image analysis and augmented reality technology,providing new ideas and methods for both fields.Through the application of multimodal medical image segmentation network and coaxial projective augmented reality stereotactic localization and visualization system,this paper explores the potential application value of medical image analysis and augmented reality technology in surgical navigation,which is a positive contribution to the future development of medical image analysis,augmented reality technology and surgical navigation systems. |
PDF Full Text Request