Medical Image Oriented Mobile Augmented Reality System

Recently,medical preoperative diagnosis systems have attracted a numerous attraction and become essential components in modern for the treatment of patients’ diseases and clinical operations,but traditional diagnosis systems have defects,such as insufficient accuracy and intuitiveness,separation of the surgical field of view and medical image,etc.,which would limit the further improvement of the current surgical quality.Mobile augmented reality(MAR)technology,with its unique advantages of combining virtual objects with the real world and good real-time human-computer interaction,has great potential for medical applications.Therefore,it has gradually been applied to medical fields of preoperative and intraoperative navigation by researchers.However,current medical mobile augmented reality systems still have certain limitations: firstly,most of the current medical MAR systems can only be applied to a specific surgical operation,which is not universal enough.Secondly,the current MAR systems on handheld mobile devices(like mobile phones,tablets,etc.)are mostly used the markerbased three-dimensional registration technology,which limits the overall convenience of the MAR system.Therefore,in this thesis,the most effective MAR Head-mounted display Holo Lens and the latest MAR systems development framework ARKit were respectively used for developing corresponding medical imaging augmented reality systems.The main work included three parts are as follows:1.The key technologies of SLAM(simultaneous localization and mapping)3D registration and the interactive design principles for mobile augmented reality systems were studied.The overall framework of medical image mobile augmented reality system was proposed.Moreover,according to different needs of medical staff,preoperative planning system based on Holo Lens and a convenient display system based on ARKit were designed,respectively.2.The key technologies of image segmentation and 3D visualization in medical image processing were studied.By using the medical image processing of open source and visualization platform 3D Slicer,this thesis implemented the segmentation and reconstruction of the patient’s original medical images and preoperative simulation planning,and finally obtained the virtual medical 3D models which could be applied to the MAR systems.3.The algorithms of virtual-real fusion and natural human-computer interaction in medical MAR systems were further studied respectively.On this basis,two sets of medical images of MAR systems was designed and developed,allowing users to realize the interactive operation of the virtual medical models through gaze,gestures and voice.