The Research Of Massive Data Processing In Disaster Monitoring

With the development of wireless sensor network related technologies, the volume and types of data collected in disaster monitoring become more huge and complex. Processing these massive data has received extensive attention. Cloud computing can enable effective processing of big data. It can realize the parallel data processing and massive data storage.The thesis proposed a system model of distributed massive data processing based on Hadoop. The system model takes the advantage of HDFS and MapReduce to support the storage and processing of massive data. This thesis focuses on data storage,processing and analysis to explore the massive data processing model design in disaster monitoring:(1) the analysis and comparison of HDFS and Swift show that HDFS is insufficient sometimes to support large data storage, while Swift has popular application in large-scale data processing. Therefore the model based on the fusion of Openstack and Hadoop for massive data processing is proposed.(2) The approach to data duplicate removal, format and data flow processing are discussed. An actor model based on HFlame architecture is introduced to realize the component for dataflow processing and data flow processing logic.(3) According to difficulties in the query and analysis of massive data, this thesis introduces Pig language to provide a simple operation and programming interface for parallel data processing.The massive data processing model is deployed on the Hadoop platform by building Hadoop cluster. Experiments show that the proposed model has higher performance in mass data processing.