Cross-modal Medical Image Retrieval

With the continuous improvement of information level in the medical industry,the amount of medical image data is expanding day by day.The current situation in the industry is that there has been a lack of effective management and retrieval methods for these multi-modal medical image data.Multi-modal data retrieval has become an urgent problem to be solved.In addition,how to quickly retrieve useful information for doctors and solve the efficiency problems in the retrieval process also need to be considered,such as lesion location,which in most cases depends on doctors to search mechanically in the retrieved sequence,and the degree of computer-aided analysis is not enough.To solve the above-mentioned problems,we have built a cross-modal medical image retrieval system,which solves the problem of how to segment and retrieve different organs contained in images of different modalities under one framework.At the same time,lesion detection is organically integrated into the process of user retrieval to help users locate lesions quickly in the retrieval results.The specific work is as follows:For the organ retrieval function,in order to highlight the contour and structure of the organ of interest to the user in the retrieval results,we need to segment the organ.On the one hand,we constructed a multi-modal multi-organ segmentation framework based on multi-task learning,unified image classification and organ segmentation under the same framework,and designed an image modal analysis and organ type reasoning module,which effectively utilized the high-level semantic information related to the organ types contained in the image provided by the user in the retrieval process,and realized the segmentation of different organs contained in different modalities.On the other hand,in the design of segmentation model,we introduce residual element in the encoder of the basic U-Net network and GAU module to replace UNet's original simple jump connection structure,to improve the basic UNet structure and realize the segmentation of different organs.From the experimental results,compared with the original UNet structure,this part of improvement has achieved better organ segmentation results.On the basis of realizing common organ segmentation,we have further completed the lesion retrieval task which is more urgent in clinical needs.In the part of lesion retrieval,we have realized the detection and classification of liver cancer,hepatic cyst,hepatic hemangioma and other common lesions in CT images of liver organs.Faster R-CNN model is adopted as the detection framework for lesion targets in the algorithm processing flow,and then the Inception v3 network is applied to the detected lesion regions through the idea of migration learning to determine the lesion categories.Compared with other commonly used classification methods,better classification results are obtained.In the process of classifying lesions,we use DCGAN network to generate image data for some categories with a small number of samples in the data set,effectively solving the problem of uneven distribution of samples in different categories of the data set,and improving the overall detection and classification index of lesions.After completing the design and implementation of the algorithm framework involved in the medical image retrieval system,we designed and implemented the application of the medical image retrieval system based on C/S architecture.On the server side,we define a number of entities and their relationship models needed in the retrieval process to facilitate our system to add and expand new retrieval target types.In the off-line processing stage,the server applies a pre-trained model to the organ and lesion retrieval data set for batch processing,and saves the processed results to the database to provide data sources for subsequent user retrieval requests.The client provides users with two ways to retrieve the organ and lesion data set of the server by using text and uploading images.At the same time,in order to enhance the interactivity of the retrieval results and serve more usage scenes such as medical teaching activities,we integrate the organ and lesion three-dimensional visualization module,render the three-dimensional structure of the organ and lesion in the retrieval results according to the segmentation results of the complete sequence,and help users to establish a more intuitive understanding of the three-dimensional structure of the retrieval target.