Research On Key Technologies Of Hot Event Mining In Social Media Stream Based On Multimodality Fusion

As one of the most important communication platforms,the explosive development of social media not only brought great influence on the language,behavior and thinking of the public,but also improved the work efficiency of government.On social media,numerous us-er-generated texts,images,videos,audios and other information,have opened a new gate to the world for the public,and a new communication channel between government and the pub-lic.The abundant data conceals countless information treasures.Mining and analyzing these treasures will have a great influence on social development.However,social media data has many unique characteristics such as big data,diversification,fragmentation,noise,etc.It is hard to use traditional methods to extract valuable information in an effective and efficient way.Thus,people cannot enjoy the benefit of information oasis,and often get lost in digital desert.To get interesting hot event for the public on evolving social media streams,based on multi-modal information fusion technology,the paper will analyze texts and images which are highly correlated to the event,and achieve the goal of hot event detection,tracking,represen-tation,recommendation and other applications.(1)The image representation of social event based on multi-modal linear fusion.As peo-ple are often misled by fake images of hot events,the dissertation proposed a novel event im-age representation model.By analyzing the fused temporal-textual consistency between event and image,it removes suspectible fake event images,and extracts representative images sam-ples that are highly correlated to the event.Though the temporal consistency between event and image can be used to detect irrelevant or fabricating event images,there is often a large error or a great shortage in the generation time of images on Internet.Thus,the dissertation designed an algorithm to estimation their generation time,based on their host documents.It uses the metadata and publish time of host documents,to estimate and optimize the image time.Generally,hot events have many scenes,and each contains many images described from different views.To represent event with image samples,based on the visual correlation of images,the dissertation uses image correlation graph to optimize the multi-modal consistency between event and image,and extracts representative images for each scene of event.The experiments show that,comparing to some state-of-the-art methods,the model can provide a better image representation of events in most cases.Besides,the simulation experiments of fabricating event images show that,the model can detect fake event images with a low-level fabrication.(2)The detection and tracking of hot event on social streams based on multi-modal deep fusion.Since social media streams evolve as the time goes by,the dissertation proposed a novel hot event detection and tracking algorithm on the stream.By annotating events with textual analysis,detecting events with multi-modal deep fusion,and tracking events with temporal smoothing,the algorithm will get coarse-grain keyword and fine-grain message de-scriptions of events.During the analyzing of event,to reduce the negative effect of massive noise on social media,the dissertation designed a new word correlation measurement,based on the importance of words and phrases.It can extract event keywords efficiently.Besides,to improve the event representation,the dissertation uses multi-modal deep learning model to fuse the texts and images of events together,and generates a multi-modal fine-grain represen-tation of events.The experiments show that,comparing to some state-of-the-art methods,the algorithm is often more suitable to detect and track social events on microblog streams.(3)The textual-visual summarization of hot event on social streams based on incremental cross-modal latent semantic analysis.To provide a vivid description of hot event streams in real time,the dissertation proposed a new algorithm for the textual-visual summarization of event.It can extract samples with strongest semantics as the representative texts and images of event after analyzing and updating the semantics of event.To improve semantic analysis of event,the dissertation designed a novel multi-modal semantics analysis model.The model will bring the main semantics of other modality to the semantics of current modality,which can improve the semantics representation of current modality.During the textual-visual sum-marization of hot event streams,the dissertation must organize the multi-modal information of event streams in an effectively way.Thus,based on pyramid time frame,the dissertation will keep the textual-visual summary of events in different granularities,according to event fresh-ness.The experiments show that,the algorithm is faster and better than other related methods,and multi-modal data fusion can greatly improve textual-visual summarization of events.(4)The content-based recommendation on social media with multi-modal heterogeneous graph.To recommend different users with latest event messages that they might be interested in,the dissertation proposed a novel social media content-based recommendation algorithm.On a large-scale textual-visual heterogeneous graph,the dissertation adopts graph clustering to optimize the semantics among entities based on link fusion,and provides personalized content-based recommendation by walking along the link of the graph.During the creation of large-scale textual-visual heterogeneous graph,to reduce time cost,the dissertation maps sim-ilar images or texts into the same block,and compute the similarity of images or texts within the block.The experiments show that,comparing to some state-of-the-art methods,the algo-rithm can provide an accurate and rich multi-modal recommendation to users efficiently.In summary,based on multi-modal fusion,this work focus on the research and realiza-tion of many hot event analysis,mining,and applications on social media streams,including the image representation of hot event,the detection and tracking of hot event,the textu-al-visual summarization of hot event and the content-based recommendation of hot event.The research achievements lay theory foundations and provide technique supports for the analysis of multi-modal fusion model and hot events on social media streams.