Research On Methods Of Multimodal Disaster Information Detection Based On Deep Learning

When a disaster occurs,people will publish all kinds of tweets in real time through social media.The tweets contain semantically rich text and intuitive pictures in disaster situations.Mining tweets containing disaster information can be used to analyze the situation of disasters,and help relevant departments to quickly make emergency decisions and public opinion analysis.With the development of artificial intelligence,the analysis of social media data using deep learning technology improves the efficiency of disaster management.At present,most related researches on disaster information inspection focus more on single image or single text methods.Data from a single modality leads to a lack of information-coupling correlations,limiting model checking performance.Multimodal deep learning,as the mainstream information analysis model in recent years,combines the characteristics of text and image information,and has a good performance effect.However,disaster analysis requires high timeliness,and high-complexity models cannot make real-time responses to disaster perception.In addition,social media multi-modal data sets for disaster detection are scarce and there are problems of unbalanced data resources and few samples.Traditional feature extraction and fusion algorithms have low data utilization.How to use limited data sets to train more universal classification models still face challenges.Therefore,this thesis mainly conducts in-depth research on the data characteristics of the two modalities of image and text in social media and how to achieve fusion tasks efficiently and accurately.Based on the general learning framework of graphic-text fusion,this thesis conducts research on disaster information on social media.In the research process,the modeling process was completed by combining relevant algorithms and models such as Transformer,attention mechanism,and associative memory neural network in the field of deep learning,so as to make full use of the useful information in the few-sample data set,and then further studied how to improve the accuracy and model speed.jointly optimized to adapt to the high real-time nature of disaster detection.Specifically,this thesis focuses on two research areas:(1)A graph-text low-rank fusion framework is proposed to detect disaster-related information on social media.In this framework,it is first innovatively proposed to extract features from different hidden layers of the model and combine them to replace the traditional final layer output,so as to capture different granularity features in text data.Each single-modal feature is processed by a lightweight attention module to reduce redundancy,and with the low-rank fusion method,the model takes into account intra-modal features and inter-modal correlations with fewer parameters.Contrast experiments,ablation experiments,and special setting experiments imitating real-world scenarios on the crisis public dataset Crisis MMD,the experimental results show that the proposed method is superior to the state-of-the-art methods in terms of accuracy and efficiency,proving that all Effectiveness of the proposed method for disaster-related information detection on social media.The proposed method has great potential in real-world applications,such as disaster response and emergency management.(2)A resource balancing model based on cross-attention and bidirectional associative memory networks is proposed to deal with the heterogeneous data modality imbalance problem that is common in multimodal data.In this model,an interactive attention module is designed to reduce redundancy and capture the correlation between modalities,and integrate the other modal correlation into this modal.Use the deep two-way associative memory module to carry out heterogeneous association,and generate a complete representation of the other party’s modality,so as to realize the completion and balance of resources.The codec module is used to perform internal association and attention on the fused representation to achieve deep balance among heterogeneous data.Finally,experiments were carried out on the crisis public data set,which proved that the resource-balanced multi-modal data used for disaster monitoring can achieve higher accuracy than single-modal and strong multi-modal methods.