Research On Automatic Annotation Technology For Cross-media Medical Images

In the age of big data,a large number of medical images are not effectively utilized.However,there is an urgent need for a large amount of annotated data in medical,education and research fields,so a technology is needed to annotate medical images.Nevertheless,manual annotation methods are time-consuming and require medical professionals to complete,the rise of artificial intelligence technology has brought a boon to the automatic annotation of medical images.Traditional automatic annotation methods only use the data of a single modality of the image,but the diagnosis report,which is closely related to the image data,is ignored.Therefore,this thesis proposes a technique for automatic medical image annotation using a cross-modal approach,focussing on how to use data from both image and text modalities for automatic annotation.The research of this thesis includes the following points:1)The author will investigate the fusion methods and fusion timing of multiple modal data in neural networks,explore the impact of different fusion methods and fusion timing on task results,and provide a ground for subsequent automatic cross-modal medical image annotation.2)The author will study the automatic labeling of disease types in medical images.Since disease type is an image-level label,this thesis will use an image classification method to study the intra-modality lesion correlation and inter-modality lesion correlation,and propose a correlation learning method based on these two correlations to better fuse the features of both modalities to improve the automatic labeling of disease types.3)The author will investigate automatic annotation of lesion locations in medical images.Since lesion location is a kind of pixel-level label,this paper will use image segmentation to study the influence of detailed information such as location,shape,size,etc.of individual lesions on the annotation,and then envisage the use of textual information to assist in segmentation based on this influence,and propose a mutual aware feature fusion method based on this idea,so that image features can better extract lesion details with the assistance of textual features,thus improving the automatic annotation of lesion location.4)Based on two of the above studies,the author have developed a cross-modal prototype system for automatic annotation of fundus images for the needs of the fundus field,which can annotate both the four common disease types in fundus images and the hyperfluorescence in fundus images,and the system provides an example of the translatability of the above studies.