Research On Key Technologies Of Modality Recognition For Images In Biomedical Literature

The amount of biomedical literatures in electronic format has increased considerably,with the rapid development of Internet.These literatures have become important public medical resources and provided massive digital medical images.The modality of a figure is an extremely useful type of metadata.It can not only help biomedical researchers and educators to understand complex biomedical concepts,but also aid to lay the foundation for archiving and indexing images in biomedical literatures,which makes more efficiently to utilize these resources.It is both time consuming andcostly to assign manually modalities to all medical images.The study of modality recognitionalgorithms comes into being to meet the strong demand for automaticallylabelingthe images.To address the greatest challenges of the diversity of modalities,the unbalancedness and scarcity of trainingsamples,this thesis leverages the techniques of image processing,natural language processing,machine learning,deep learning and transfer learning to efficiently and accurately recognize biomedical modalities from images or captions in biomedical literatures.Recognition involves modality classification of simple images and multi-label classificationof compound figures after detecting compound figures.Therefore,compound figure detection,modality classification and multi-label classification are studied in this thesis.The main contents of this thesis include the following three aspects:For the detection of compound figures in biomedical literatures,cross-media compound figuredetection model based onconvolutional neural network(CCFD_CNN)is proposed.The model learnseffective representations from visual content and textual information to detect compound figuresthrough convolution and pooling operations of convolutional neural network(CNN).Compared to current hand-design method,CCFD_CNN needs less feature engineering that enhances its generalization ability.CCFD_CNN achievesthe state of the art performance on the compound figure detection problem(ImageCLEF dataset).For the modality classification of simple images in biomedical literatures,visual ensemble model for modality classification(VEMMC)is proposed.VEMMC assembles CNNs with three different depths to capture diversity feature of biomedical image modalities,by means of fine-tuned transfer learning and data augmentation method to relieve the overfitting problem and training CNNs with six weight layers from scratch to capture more biomedical features.VEMMC obtains satisfactory performance on the modality classification problem(ImageCLEF dataset).For the multi-label classification of compound figures in biomedical literatures,cross-media multi-label classification model based on hybrid transfer learning(Hybrid_TL_CMC)is proposed.Hybrid_TL_CMC extracts modality information from visual content using deep CNNs,by means of learning general features from natural images and more domain features from biomedical simple images and their captions.This kind ofinethod combining heterogeneous and homogeneous transfer learning is designed to relieve the problem of overfitting to the majority class leaded by an unbalanced distribution of the labels.Visual and textual informationareincorporated into the model through two-step fusion strategy,Hybrid_TL_CMC achieves better performance on multi-label classification problem(ImageCLEF dataset).