Ultrasound Image Caption Based On Object Detection

Medical image is a kind of digital image which is widely used in clinical diagnosis,but it has the characteristics of low resolution and fuzzy boundary between different organs and tissues.In recent years,the research on automatic semantic understanding of medical image has become attractive.Object detection and image caption both are important branches of computer vision,and they have made a remarkable development in recent years.It is of great clinical application value to combine these two algorithms and introduce them into medical image processing.The location of the lesion area in the medical image can be detected by the object detection algorithm,and the lesion area is encoded separately,which makes the coding vector carry more useful information,and eliminates the interference of the background and unrelated organs in the medical image,so as to improve the quality of the diagnosis report.The existing medical image caption algorithms focus on improving the similarity between the generated report and the ground-truth report,but the accuracy of pathological information of generated report is not considered,which reduces the reliability of the generated report.Therefore,based on the object detection algorithm and the image caption algorithm,this thesis proposes two medical image caption algorithms to improve the quality of generated diagnostic report and the accuracy of pathological information in the generated report: one can integrate pathological information,and the other one can utilize reinforcement learning to train the model.The introduction of the research work in this thesis is as follows:1.In order to improve the problem of inaccurate pathological information in the diagnosis report generated by the current medical image caption algorithms,this thesis proposes a model which can fuse pathological information to generate medical image caption.The model contains an object detection model to detect the lesion area of medical image,then encodes the lesion area and extracts the pathological information.Decoding model learns to fuse image coding vector and pathological information to generate diagnosis report.Experimental results show that the pathological information in diagnosis report generated by the proposed model is more accurate than the models only input visual information.2.In order to utilize the samples that have pathological information errors in generated diagnosis report to optimize the parameters in report generation model,and this thesis proposes a medical image caption model based on reinforcement learning.Adding a pathological diagnosis branch to the report generation model,and the loss is obtained by comparing the predicted results with the ground truth,then policy gradient algorithm of reinforcement learning is used to calculate gradient and optimize parameters,which make the pathological information more accurate.3.In this thesis,a simulation system for automatic generation of medical image diagnosis report is designed and implemented,and the system integrates the two models proposed in this thesis.Users can input ultrasound image or X-ray image into the system by simple operation,after some super parameters has been settled,the system can automatically generate the diagnosis results of medical image.The system not only provides an expandable platform for future research,but also has a potential clinical application value.