Research On Rumor Detection Based On Optimizing Multimodal Model Structure

Nowadays,there are increasingly more people consuming news through social media,which can provide timely and comprehensive multimedia information on the events taking place all over the world.The content of online news is usually composed of text content and visual content.Compared with the traditional news,the online news with images and videos can provide better narrative effect and attract more attention from readers.At the same time,rumors containing false or even fake visual content have proliferated and spread widely,causing massive negative effects and even manipulating public events,which has become a major problem related to the public and the government.Therefore,we focus on researching the detection methods of network rumors with various modal contents,and the relevant contents are as follows:(1)Research on feature extraction based on multimodal network rumor.Aiming at the content form characteristics of multi-modal network rumors,multimodal feature learning is carried out for text content and visual content based on deep learning.Among them,for text content,a gate structure focusing on the importance of input elements is added to the LSTM,and the Ele Con G-LSTM is proposed to make the neuron have the attention ability to adaptively focus on the important element of input,finally compare Ele Con G-LSTM with other basic neurons with experiments;For visual content,from the perspective of visual objects,adopts the Faster-RCNN model to capture multiple visual object features and compares it with the global visual features extracted by the VGG-19 model.The results show that the Ele Con G-LSTM effectively pay attention to the contributions of different elements in the input information,improve the efficiency of time series modeling,and can be effectively applied to the basic construction of feature extraction model.For visual content with rich features and complex combinations,the regional object features extracted based on Fasters-RCNN can provide more comprehensive and effective information compared with the global visual features extracted from VGG-19.(2)Research on rumor detection based on multimodal feature.Based on the effective extraction of text-visual features,a multimodal hybrid fusion network AHFNN based on attention mechanism is proposed for rumor detection by researching the fusion methods of two different modalities at the word-visual object level.Using Ele Con G-LSTM and Faster-RCNN to extract feature matrices about words and visual objects respectively,build a modality feature fusion module based on the attention mechanism to complete the high-level interaction between words and visual objects,and proposes Adaptive-SA on the basis of self-attention,which constrains the flow of information within the modality,and finally the visual features updated by the word feature and the global sentence vector extracted by BERT are fused with adaptive training weights for rumor detection.The model is compared with the rumor detection results of the single-modality and multimodal rumor detection models.The results show that Adaptive-SA can make the relationship modeling within the modal more targeted and diverse,and AHFNN can reasonably handle the fusion of multimodal features,the performance of rumor detection is significantly improved compared with other baseline models.(3)Structure optimization of multi-modal rumor detection model based on DARTS.Based on the AHFNN for multimodal rumor detection,neural architecture search algorithm is used to automatically search the structural composition of modality feature fusion module in AHFNN.The pool of operations to be selected includes Adaptive-SA,cross-Attention and full connection layers,and uses gradient descent method to encode visual features with text features.Comparing the model structure of automatic search with the rumor detection results of AHFNN and other baseline models,the results show that the optimal model structure automatically searched by the neural architecture search algorithm achieves the best detection performance.