| Medical report,usually provided by specialized radiologists,is a textual description of the corresponding medical image,which is an important way of communication between radiologists and referring physicians.But faced with a large number of patients,writing reports for all the images one by one takes a lot of time for doctors.Besides that,the difference in experience of doctors also makes the abnormal information in some medical images ignored,resulting in the inability to generate accurate medical reports.Considering these problems,the rapid,accurate and automatic generation of reports with the help of Artificial Intelligence(AI)has important practical value for improving the work efficiency and service quality of hospitals.Therefore,medical report generation has become an important topic in the research field of intelligent medical in recent years.The purpose of the medical report generation task is to automatically understand the content of medical images and provide accurate descriptions to facilitate clinical decision-making.It is a task combining Computer Vision(CV)and Natural Language Processing(NLP),involving multiple technologies such as medical image recognition,medical image segmentation,text generation,and computer-aided diagnosis.Different from the image caption task,medical reports highlight the relationship between medical imaging objects and clinical findings,making it an extremely challenging task.This paper studies and analyzes the existing medical report generation methods,and constructs a new medical report generation method.The main research work of this paper is summarized as follows:(1)Although a lot of work aims at medical report generation tasks,there are still some challenges.First,the descriptions of multiple organs and lesions and their spatial location relationships in medical reports are still not sufficient and accurate,and the use of single-scale feature extraction methods may overlook much important information.Second,most existing methods generate reports with repetitive words or lack of key words.To solve the problems mentioned above,this paper proposes a medical report generation method Multiscale Feature Extraction and Word Attention Based Network(MFWAN).At first,the model incorporates a multi-scale feature extraction module,which uses spatial information at different scales of medical images to focus on abnormalities in different regions and establish long-term associations at the channel level.Then the model uses a multi-classification contextual information generation module to classify visual features and generate contextual semantic information.At last,the report generation module based on the word attention mechanism generates semantic vectors and word vectors in different stages,assigns different attention to different word vectors,generates more accurate words that imply key information about the disease,and finally generates a complete medical report.(2)In order to solve the problems that the MFWAN model generates reports with single words while lacking specialized terms,this paper further proposes Retrieval-based MFWAN(Re-MFWAN),an optimized retrieval method for medical report generation based on crossmodal feature fusion.This method utilizes an image-report retrieval module to find the most matching report for the current medical image,forming a report template to provide doctors with a basis in diagnosis;the report-sentence retrieval module takes the correlation between report contexts into account to find the next most relevant sentence for the current one.Different from MFWAN which takes medical image features as the input of the decoder,Re-MFWAN fuses the image features of medical images with the retrieved text features as the input of the decoder and uses similar reports as a corpus of templates to increase the accuracy and professionalism of the generated reports.The experimental results in IU X-Ray dataset prove that compared with other state-of-theart models,the proposed method can automatically generate more accurate,coherent,and professional medical reports based on the medical images of patients,and has good value of clinical application. |
