Design And Implementation Of Social Media Disaster Information Mining System Based On Distributed Crawler

In recent years,our country's social media has developed rapidly.Social media such as Sina Weibo generates a large amount of data every day.How to improve the efficiency of social media data collection and mine disaster-related information,especially time and space information,for disaster information Decision support for management and disaster relief and prevention is of great significance.This paper proposes a social media disaster information mining system based on a distributed crawler,which collects data efficiently through a multi-node distributed mode,and extracts the temporal and spatial feature distribution information in it to provide data support for disaster management.The disaster information mining system uses the B/S model and adopts a four-tier architecture design of data collection,data storage,data mining and data display.The data collection layer is based on the Scrapy-Redis framework,uses the four Cent OS servers deployed on the Alibaba Cloud server as crawler sub-nodes,uses the Sina Weibo platform as the data source,and uses anti-crawler technology as an auxiliary means to capture data.The data storage layer includes two steps.First,the original Weibo data is cleaned and stored in the My SQL database.The data mining layer is to conduct disaster information mining on microblog data,mainly to extract time feature information and spatial feature information,time feature information distribution database query technology to extract,microblog text place name recognition uses place name database matching and place name prefix and suffix algorithm At the same time,and with the help of the Gaode map development interface,the inverse coding is converted into latitude and longitude coordinates.The data display layer is to visualize the collected Weibo information and the data information after analysis and processing in the form of charts and other forms.The webpage built by the Django framework is the basis for data display.Weibo text information,Weibo user information and processing analysis The time characteristic information of is displayed in the form of charts using the ECharts library,and the spatial characteristic information is displayed using the Gaode map display location distribution.Through the four-layer architecture design,the efficient capture of social media data,disaster information mining and visualization are completed.The disaster information mining system uses the Lekima and Bailu typhoon disasters with different occurrence times and different impact ranges as cases.It captures relevant data on Sina Weibo and excavates time and space characteristics,and then conducts time distribution analysis and spatial distribution analysis.Experimental results show that the data collection time of each disaster can be kept within 20 minutes,and the temporal and spatial distribution can reflect the evolution of typhoon disasters.The social media disaster system based on distributed crawlers can efficiently capture social media data and dig out disaster information from it.It can be applied to a variety of disaster scenarios including typhoons and provides a new method for disaster information management.