| Heart Failure,or HF,it is a complex clinical syndrome that occurs mostly in the end stage of heart disease.It is a disease with high morbidity,mortality,and healthcare costs.With the increasing global aging,heart failure has become one of the world’s leading public health safety issues as it places a heavy burden on global health care systems.Traditional clinical diagnosis based on careful history taking and physical examination has imposed a heavy workload on medical personnel.It has been a major challenge to effectively assist in the diagnosis of heart failure patients and help physicians predict their risk of morbidity.The development of electronic medical records and deep learning has provided technical support for heart failure assisted diagnosis.Traditional computer-aided diagnosis usually uses independent clinical data or an independent model to analyze and diagnose the risk of heart failure.However,this method ignores the large amount of disease association information in electronic medical records,may result in low accuracy of heart failure diagnosis.Therefore,this paper investigates the method of heart failure assisted diagnosis and proposes a method of heart failure assisted diagnosis based on multimodal data fusion and multi-model integration.First,13746 cases of experimental data that can be used to study heart failure assisted diagnosis were selected from the MIMIC-III public database.Patients with heart failure admission death and readmission were screened according to experimental needs.Clinical notes,structured demographic characteristics and laboratory test data(including patients’ vital sign data)of heart failure patients were extracted to integrate multiple sources of information to construct multimodal input data.Then,data pre-processing is performed on the multiple raw data collected.Secondly,the time-series characteristics of clinical notes are combined to build an attention mechanism-based Bidirectional Long Short-Term Memory model for heart failure assisted diagnosis.The model addresses the problems of poor extraction of long-distance semantic features and partial information loss by the Bidirectional Long Short-Term Memory.The attention mechanism is introduced to improve the effectiveness of feature extraction.Meanwhile,the model integrates all the information of clinical notes and assigns different attention to the current disease diagnosis information,so that the model can predict the risk of heart failure disease more accurately and effectively.Finally,a heart failure assisted diagnosis method based on multimodal data fusion and multi-model integration is proposed.Using the characteristics of diversity and heterogeneity of electronic medical records,the unstructured data(clinical notes)and structured data(demographic characteristics,laboratory test data,etc.)of electronic medical records are combined as multimodal input sources.Multiple deep learning models(Bi LSTM,Bi LSTM+Attention,CNN,CNN+Bi LSTM)are trained separately to obtain multiple observation perspectives,and the final prediction results are obtained by learning and weighting each prediction result by linear regression method.In order to improve the prediction tasks related to the auxiliary diagnosis of heart failure,the prediction tasks of admission mortality,30-day readmission and readmission risk prediction for heart failure patients were added at the later stage of the experiment.The heart failure ancillary diagnosis model proposed in this paper is trained and evaluated in the MIMIC-III public database.Experiments show that this method can complete the risk prediction tasks related to heart failure accurately and effectively.Especially for the auxiliary diagnosis of heart failure the area under the ROC(Receiver Operating Characteristic Curve)curve AUC reached 95.91%.Moreover,this method can be applied to other heart failure ancillary diagnostic risk prediction tasks.It indicates that multimodal data fusion and multi-model integration methods can obtain far more information than single data.And it provides a new way to take full advantage of the diversity of electronic medical record data. |