| Minimally invasive surgery(MIS)has the advantages of less trauma,less blood loss and faster recovery.With the deep integration of the new generation of information technology and the medical industry,MIS has begun to evolve towards intelligence.The chief surgeon,auxiliary doctors,operating room nurses,and other medical staff,as well as medical equipment such as intelligent endoscopes and intelligent medical robots,form a human-machine hybrid surgical team.To successfully deploy MIS,the humanmachine hybrid surgical team works together throughout the procedure.As a result,operating room management has changed from traditional behavior management for the doctor team to behavior management for the entire human-machine hybrid surgical team,and intraoperative behavior decision-making has switched from traditional manual behavior decision-making to human-machine hybrid behavior decision-making.The ability to set up behavior management and decision support for the human-machine hybrid surgical team has emerged as the determining factor for the quality and effectiveness of MIS.This thesis focuses on the MIS scenario where the surgeons,nurses,an intelligent endoscope,and an intelligent robotic endoscope holder cooperate with each other.Research on the intelligent perception and behavior decision-making methods of the human-machine hybrid surgical team in this scenario.Specifically,the methods include intelligent perception,interaction recognition and assisted decisionmaking,and human-machine collaborative behavior decision-making.In this thesis,in vivo environment perception ability of humans and machines is enhanced by intelligent analysis of endoscopic images.Then,surgical interaction recognition and assisted decision-making based on surgical environment perception ensures that the human-machine hybrid surgical team has a shared understanding of the present surgical stage,behavior,and aim.The following stage of surgical behavior decision-making is guided by future surgical interaction prediction method based on endoscopic video.Finally,the surgical behavior decision-making method of the robotic endoscope holder is established for man-machine collaborative control,which greatly improves the behavior decision-making efficiency and collaboration ability of the human-machine hybrid surgical team.The key scientific issue of this thesis is intelligent analysis of surgical scenes and human-machine collaborative behavior decision-making based on endoscopic visual perception.The specific scientific problems are:(1)How to reconstruct a threedimensional(3-D)in vivo surgical scenario using limited endoscopic visual perception data,providing surgeons greater stereoscopic and intuitive visual perception abilities.(2)How to extract surgical operation patterns and rules to better understand and forecast surgical interactions,providing intelligent surgical process guidance to the surgical team.(3)How to establish a human-machine collaborative behavior decision-making mechanism,providing surgeons with safe and convenient robot assistance.The following is a list of the most significant contributions.(1)An unsupervised 3-D perception enhancement method based on continuous endoscopic video is proposed for MIS safety assurance.In vivo surgical environment perception is the basis for analyzing surgical behavior and making surgical decisions.Currently,the images of in vivo environment are collected and presented through an endoscope during MIS,which only provides surgeons with a two-dimensional image information perception channel,and cannot provide accurate 3-D spatial information for surgical behavior analysis and decision-making of human-machine hybrid surgical team.Existing unsupervised depth estimation research fails to make full use of the common feature information of endoscopic video and assumes inflexible deformation between consecutive frames,the performance and efficiency of the method can be further improved.Based on the existing unsupervised estimation methods,the unsupervised depth estimation method is proposed in Chapter 3.It innovatively proposes a space-time cross correlation mechanism of continuous frames and a multitask learning framework.Integrating the spatiotemporal correlation mechanism of consecutive frames and considering the depth estimation results of consecutive frames can further improve the performance of depth estimation.Furthermore,multi-task learning makes full use of the general visual features of endoscopic images,further reducing the number of network parameters and improving the efficiency of the depth estimation network.(2)A fine-grained surgical behavior management and decision-making method based on prior knowledge and attention mechanism is proposed.Currently,there is a lack of unified real-time management and planning for surgical behaviors during the MIS.Relying on the manual instructions of the chief surgeon has the drawback of delaying cooperation and difficulty in preparing for future surgical behaviors in advance,which limits the improvement of the quality and efficiency of MIS.Existing research rarely involves fine-grained surgical interaction recognition and future interaction assisted decision-making.The fine-grained surgical interaction includes "instrument,verb,target" triplet.Existing research is insufficient to assist the humanmachine hybrid surgical team in making decisions on current surgical behavior,stages,goals,and future behavior cognition support.The surgical behavior management and decision-making method based on prior knowledge proposed in Chapter 4 innovatively integrate surgeons’ clinical personalized clinical experience into the online recognition and future prediction model of surgical interaction triplet.The attention mechanism learns the complex relationship between triplets and within triplets and improves the accuracy of online recognition of surgical interactions.The transformation rules between the surgical interaction triplets can be obtained by mining the surgeon’s personalized operations from clinical data.The fusion of prior clinical knowledge and the spatial-temporal features of historical endoscopic videos further improves the m AP of future surgical interaction triplet assisted decision-making.(3)A human-robot collaborative in vivo 3-D collision avoidance and field of view(FOV)tracking behavior decision-making method for robotic endoscope holders is proposed.On the basis of environment perception enhancement and unified cognition of surgical behavior,the human-robot collaborative behavior decisionmaking method of the robotic endoscope holder can be formulated to ensure MIS safety and human-robot cooperation ability.The existing research on advanced intelligent robotic endoscope holders does not consider the surgeon’s surgical behavior,intention,and the 3-D in vivo working environment when performing adaptive FOV adjustment,which reduces the tacit understanding and safety of man-machine cooperation.The human-robot collaborative behavior decision-making method in Chapter 5 fully takes into account the clinical requirements of safety and different instrument concerns.Based on the traditional visual servo model,the visual tracking vector and the 3-D collision avoidance vector are innovatively designed to meet the constraints of incision points.The visual tracking vector automatically adjusts the endoscopic FOV by tracking the instruments in the endoscopic screen with different weights,can adjust the insertion depth adaptively,and is not affected by the entry and exit of the instruments.In addition,the 3-D collision avoidance vector can effectively prevent the robotic endoscope holders from actively colliding with the instruments operated by the surgeons and accidentally touching the body tissue,etc.,to ensure safety.The research in this thesis enriches the theory of intelligent perception and behavior decision-making for human-machine collaborative MIS.It has rich practical implications for reducing the uneven distribution of medical resources and insufficient overall quantity of medical resources in China.It also has significant managerial implications for boosting the intelligent management level of MIS operating rooms,improving the quality of medical services,and patient satisfaction.Future research work will focus on intelligent auxiliary decision support during the entire surgical operation,as well as multi-objective surgical robot behavior decision-making and optimization method. |
PDF Full Text Request