Research On Cross-modal Hash Retrieval Technology For Chest CT Image-text

With the rapid development of network technology in the field of medicine and the gradual improvement of medical information storage standards,medical data is growing fast.In addition,due to the characteristics of medical data,it takes on the form of multiple modes,and different modes often appear simultaneously and supplement each other.In the face of large-scale medical data with different modes,how to establish the correlation between them,make full use of the semantic information of different modal data,and realize the mutual retrieval of different modal information has important application value for the medical field.The incidence of cancer is the highest in our country,radiologists detect early lung cancer in the form of chest CT image detection and screening nodules,and save the test results in the form of text,as the diagnostic basis for clinicians.Due to the large number of chest CT images,and the location and size of pulmonary nodules in the lung are not fixed,the diagnosis of pulmonary nodules pays more attention to finegrained information,which brings heavy workload to radiologists.Above these problems,this paper proposes a cross-modal retrieval method for chest CT image and text.The proposed cross-modal retrieval method can fully explore the high-level semantic information between the two modal data,retrieve the corresponding diagnostic information through the marked pulmonary nodules,or retrieve the corresponding pulmonary nodules image through the textual description information of pulmonary nodules.In this paper,the cross-modal retrieval model of chest CT image and text is introduced in two steps.The first step is the feature extraction of CT image and text,and the main job is the fine-grained features extract of CT image,and the second step is to learn hash coding by using extracted image features and extracted features of corresponding text.Finally,the cross-modal retrieval between CT image and the text is realized.As the first step,because pulmonary nodules on chest CT images in the position and size are no rules to follow,to deal with this problem,based on labeled data with a good sized accordingly pulmonary nodules of cutting,the purpose is to extract more accurate pulmonary nodules of high-level semantic information,reduce other organs for pulmonary nodules in the lung impact of feature extraction.Pulmonary nodules usually exist in a three-dimensional form.Traditional methods for extracting and classifying characteristics of pulmonary nodules are often based on single-layer pulmonary nodules.Although pulmonary nodules are spaced at the millimeter level,there are still some differences in the fine particle size between adjacent sections for pulmonary nodules with the maximum length and diameter of only 30 mm.Therefore,in order to solve this problem,this paper adopts multi-level and second-order fusion feature extraction method to extract the feature information of pulmonary nodules.This method extracts more complete feature information of three-dimensional pulmonary nodules based on the features of single-layer fine-grained pulmonary nodules.This method evaluated the AUC values at 0.92,0.7 and 0.87 demonstrating their effectiveness.For the second step,since there is often a large semantic gap between different modes,the solution of cross-modal retrieval is to map the characteristics of different modal data into a public space,and achieve the goal of narrowing the semantic gap by constraining the two types of modal information in the public space.In this paper,the feature information of pulmonary nodules image extracted in the previous step and that of corresponding text are mapped to hamming space,and a similarity matrix based on data sample category annotation is constructed to constrain the obtained hash code.Through experiments on the LIDC data set,the MAP of text and image retrieval in the experimental results reached 73.26% and 72.39%,respectively,indicating the effectiveness of the supervised semantic association method between heterogeneous data and the feasibility of the cross-modal retrieval system of chest CT image and diagnostic text.