Research And Application Of Dynamic Emergency Monitoring In Real-time Streaming Big Data Environment

Emergencies,that is,unexpectedly major sensitive events,not only affect people's daily lives,but also cause serious social harm.For the reporting of emergencies,traditional news media need to ensure the accuracy and authenticity of information,and there are often lags.Sina Weibo,as a communication platform for sharing,disseminating,and obtaining short real-time information,has a large number of mobile online users,which guarantees instant reporting of emergencies,and it is easier to form social topics and cause intense discussions.In order to better respond to emergencies,it is of great significance to monitor emergencies from the massive information flow.Through accurate and timely reporting of emergencies,relevant departments can quickly take countermeasures to avoid causing social panic.The public can understand the nature of the emergency and take appropriate measures in advance.Based on the above analysis,after investigating and studying emergency monitoring technology and clustering related literature,using the efficiency of Storm distributed computing framework in processing real-time streaming data,facing the constantly generated microblog data stream,an efficient online distributed emergency monitoring model is proposed.The model first uses Kafka for reliable transmission of pipeline data streams,and uses the time window mechanism to operate the data in the time window under the Storm framework to achieve continuous monitoring of emergencies.First,filter and analyze the text data stream through appropriate measures;then,improve the keyword selection weights,complete the optimization of the emergency event monitoring model,and realize the extraction of sudden words per unit time.Finally,to generate a cooccurrence matrix based on co-word analysis,normalize it to a similarity matrix with the help of Jaccard coefficients,improve the condensed hierarchical clustering algorithm,complete the identification of the similarity between clusters during the clustering process,and realize online streaming clustering Class,select the appropriate cluster,and complete the emergency monitoring under the time window.This article uses crawler technology to collect about 40 w of data on January 8th and 9th,2020,and uses Kafka's regular transmission to generate real-time data streams,monitor emergency events,and communicate with emergencies and traditions published on the Internet.The method uses agglomerative hierarchical clustering to compare and verify the feasibility and efficiency of the model.The test results show that the emergencies monitored by the model coincide with the emergencies reported by the Internet by 83%,and the keyword clusters of emergencies are generated to describe the emergencies effectively,and the emergencies under the time window are realized.Real-time monitoring of events.While improving the real-time performance of monitoring,the accuracy of monitoring is ensured,which provides a valuable reference for real-time event monitoring under massive real-time data streams.