Research On Brain CT Medical Report Generation Based On Weakly Guided Attention Mechanism

Brain computed tomography(CT)imaging reports are a routine procedure for diagnosing cerebrovascular disease.However,writing radiological image reports has become a serious workload for radiologists.Automatic report generation can alleviate radiologists’ workload and reduce the diagnostic errors,thus achieving the purpose of assisting clinical diagnosis.Automatic Brain CT report generation has important research significance and research value.At present,the research on automatic generation of brain CT medical reports is still in the initial exploratory stage and this task faces the following challenges: First,Brain CT lesions are dispersed in 3-D space,with more morphological instability.It is difficult to identify lesions and extract effective visual features.Second,the Brain CT reports are long paragraphs with the similar medical terms and irregular structure.It is difficult to train the language model.These challenges increase the difficulty of lesions recognition and report generation for the Brain CT imaging.In particular,the attention model has the ability to extract key features from mess visual information.The attention-based report generation methods have also become a current research hotspot.However,Brain CT report generation requires more concentration on importance slices and tiny lesion areas,which is hard to recognize and learn by the current attention.To cope with the shortcoming,this thesis has completed the following two research works:(1)In order to guide the attention mechanism to capture important lesion areas better in Brain CT and generate more accurate Brain CT medical reports,this thesis proposes a brain CT medical report generation method based on a weakly guided spatial attention mechanism(WGSAM)following the encoder-decoder framework.Specifically,in the encoder,a two-layer attention model(TAM)is designed to extract the visual features of Brain CT images.First,weakly guided spatial attention mechanism is used to capture features of brain lesions in each slice where the activation maps of lesions generated by Grad-CAM are taken as the weak labels of lesions to guide the attention learning.Then,frame attention is used to learn important brain CT slice sequence features adaptively.In the decoder,a keyword-driven interactive recurrent network model(KIRN)is designed as a language generation module to generate a brain CT medical report.First,the possible keywords denoted positions of lesions is used to activate the initial hidden layer state of the language model.Then,we improve the accuracy of brain CT medical report generation through the dynamic interaction of word LSTM model and sentence LSTM model.The results on our constructed brain CT medical report dataset show that the proposed method can effectively guide the attention mechanism to learn important lesion area features in each brain CT slice,and improve the performance of brain CT medical report generation.(2)In order to solve the difficulty of adaptive sequence attention mechanism to learn the correspondence between brain CT slice sequences and sentences,this thesis proposes a hierarchical weakly guided attention mechanism(HWGAM).HWGAM includes a hierarchical weakly guided attention mechanism and a hierarchical language model.Specifically,the hierarchical weakly guided attention mechanism consists of a weakly guided frame attention model and a weakly guided spatial attention model,which are used to capture the features of important slice sequences and lesion regions in a single slice,respectively.First,prior knowledge is used to guide the model to learn the correspondence between CT slice sequences and sentences in the weakly guided frame attention model.Then,the Grad-CAM algorithm is used to guide the model to learn the correspondence between brain CT image regions and words in the weakly guided spatial attention model.The hierarchical language model consists of a sentence LSTM model and a word LSTM model,which are used to generate sentence vectors and word vectors,respectively where sentence LSTM and weakly guided frame attention model,word LSTM and weakly guided spatial attention interact dynamically and improve each other.The experimental results show that the method proposed in this thesis can effectively guide the hierarchical attention mechanism to learn the alignment between images and sentences,image regions and words,and improve brain CT medical report generation performance.This thesis studies and explores the attention mechanism in the automatic generation of brain CT medical reports from the perspective of weakly guided attention mechanism.The work completed in this thesis not only enriches the research on brain CT medical report generation methods,but also promotes the application of attention mechanism in brain CT imaging.