Computer-aided Surgical Tool Detection Algorithm And Surgical Workflow Recognition Based On Deep Learning

With the continuous improvement of modern medical technology,surgery has entered a revolutionary era and it is developing towards minimally invasive and digitalization.Computer-assisted surgery(CAS),as the initial revolution of digital minimally invasive surgery(MIS),has been widely active in the forefront of surgical treatment.At the same time,surgical tool detection and surgical workflow recognition are important research issues in the scope of CAS.In a computer-assisted MIS system,the surgical tool detection algorithm can provide doctors or auxiliary robots with accurate and real-time information,such as trajectory and position information,and then help the surgeon to formulate the best surgical planning or advance surgical warning,thereby reducing patient complications and making surgery safer.In addition,the recognition of the surgical workflow during the surgery is beneficial to improve the operation efficiency and assist less experienced surgeons to make correct decisions.Moreover,the automatic recognition of the surgical workflow for the complete surgery video is helpful to the skill assessment of surgeons and the indexing of surgical video databases.However,most traditional surgical tool detection algorithms have the disadvantages of insufficient accuracy,time-consuming,expensive,and large space in the operating room.Therefore,the emergence of surgical tool detection algorithms based on deep learning has added new impetus to the development of CAS.However,it is a pity that the current surgical tool detection algorithms cannot achieve both accuracy and speed.At the same time,very few researchers pay attention to automatic surgical workflow recognition that helps standardize surgical procedures.Based on actual needs,this article mainly designs a fast and accurate surgical tool detection algorithms and surgical workflow recognition method based on deep learning.The main work is as follows:(1)Set up a dataset in real surgical scene:Most of the surgical tool detection algorithms proposed by the current research are based on the ATLAS Dione and EndoVis challenge datasets.The video of the former is a simulated surgical scene;the latter is a real surgical video,but the moving speed of the surgical tool is very slow and cannot represent the complicated situation in conventional surgical scenes.Therefore,we manually annotated the videos of the Cholec80 dataset,and obtained a new Cholec80-locations dataset with positioning information.This dataset is obtained in a complicated real surgical scene.(2)Design two detection algorithms of surgical tools:In this paper,a one-stage lightweight anchor-free convolutional neural network model is designed,and the residual model in the original Hourglass network is replaced by the classical fire module.At the same time,the boundary box of the surgical tool is obtained through the center point of the surgical tool without post-processing.Comparative experiments show that this method greatly reduces the parameters needed for training the convolutional neural network model,and meets the requirements of accuracy and real-time in the process of detecting surgical tools,which is obviously superior to other surgical tool detection algorithms based on deep learning.On the other hand,we designed a surgical tool detection algorithm based on convolutional long and short-term memory and convolutional neural network.This algorithm combines the multi-scale idea and utilizes the spatio-temporal information of images to complete the task of surgical tool detection quickly and accurately.By processing the output information(existence,category,location)of the detection algorithm of surgical tools,we can obtain the use information(heat map,action timeline,motion trajectory map,etc.)of surgical tools in the surgical video to provide data support for evaluating the surgical skills of surgeons.(3)Propose an automatic surgical workflow recognition algorithm:Since detecting the types of surgical tools in the current video frame is one of the bases forjudging the current surgical workflow,this paper further proposes an end-to-end trainable recognition algorithm for extracting the spatio-temporal information of video images to output the surgical workflow.The algorithm adopts residual network and independent recurrent neural network,which is a double-branch structure that outputs the existence information of surgical tools and the stage of surgical workflow.We compared and analyzed various surgical workflow recognition algorithms through evaluation indexes such as recall,F1 score,confusion matrix,etc.Experimental results show that this method is better than the network model using long short-term memory network,and the recognition result is more accurate and efficient.