Design And Implementation Of Hadoop-based Environment Quality Data Repair System

With the development of industrialization,People’s life is troubled by the increasingly serious environmental pollution.In order to strengthen the monitoring of urban environmental pollution,large scale infrastructure construction of environmental quality monitoring stations has been carried out in key cities.With the rapid growth of environmental monitoring data,the efficient storage of data cannot be satisfied by the current data storage methods.The current data storage methods are not conducive to efficient data storage,which leads to frequent database data loss and abnormal problems.In recent years,Hadoop distributed storage technology has become a new way to efficiently store environmental quality monitoring data and solve the problem of "information island".This research is based on the Hadoop framework,this paper designs a big data Environmental monitoring system,the Environmental monitoring system can not effectively deal with large amounts of data,improper storage of environmental quality data and low reliability of environmental quality data restoration.The specific tasks are as follows:Firstly,according to the development status and existing problems of big data and data repair algorithms at home and abroad,the key technologies of constructing environmental quality monitoring platform are expounded,including Hadoop framework,data acquisition and pre-processing,and data repair and repair algorithm,which lays a foundation for the design and research of environmental quality monitoring platform.Secondly,the overall architecture design of environmental quality monitoring system based on big data technology was proposed.The detailed design research was carried out for each functional layer of the system.From the data acquisition,data storage,data analysis and data application.On the basis of the theoretical support of big data technology and the research of current environment quality monitoring system,the design of Hadoop Big Data Platform is completed,which is based on Environmental monitoring.Thirdly,on the basis of the overall architecture design,the Hadoop big data Environmental monitoring platform is built,which provides a guarantee for the environmental quality data restoration.Aiming at the non-stationary and nonlinear characteristics of the data of the Environmental monitoring platform,an environment quality restoration model based on Median filter(MF),empirical mode decomposition(EMD)and integrated moving average autoregressive model(ARIMA)is proposed,in order to improve the repair accuracy,the pretreatment and repair tasks are combined.Finally,after the system test hardware environment,software environment,and software program have been deployed,functional tests were conducted on the data acquisition layer,data file storage layer,Data pre-processing layer,mixed storage layer,data access layer and data application layer of the environmental quality monitoring system,and the performance test of data query and data import,so as to perfect the system function and improve the system performance.This paper focuses on the construction of environmental quality monitoring system of big data,and the research of environmental quality data repair model,The experimental data show that the model can effectively improve the repair accuracy of environmental quality data.The improvement of Mae,RMSE and RMSE is 67.1%,61.1% and 65.3%,respectively,it provides a reliable database for Hadoop-based environmental quality repair system and a reliable guarantee for environmental pollution prevention.