Based On Spatial-temporal Correlation Sensory Data Cleaning Research

As one of the carriers of civilization development,data has a vital position.Throughout the history of the development of civilization,it can be found that accurate information and data can even determine the prosperity and decline of the country.With the landing of 5G communication,the promotion of IPv6 protocol and the development of blockchain technology,we are now in a new round of technological information technology revolution.In the Internet of Things,hundreds of millions of sensor devices collect huge amounts of data every day.Not only that,but these data volumes are still growing exponentially.Among the raw data collected by massive sensors,there is a part of redundant,conflicting,abnormal or missing data,which is dirty data.Dirty data can only express the original data completely and accurately after being properly cleaned,better serve the data manager,support its decision-making and analysis,and play the true value of the data.In this paper,by establishing a data cleaning framework and designing data cleaning methods,four types of dirty data in the perceived data are cleaned.Through the comparison of the accuracy of different cleaning models,it is obtained that in the data set of temperature,humidity,carbon dioxide,and light intensity,the data is cleaned through the framework designed in this paper,and the accuracy is optimal after filling in with the space-time correlation model.In the research process,this article mainly designed the following five aspects.To begin with,the design of the data cleaning framework.According to the needs of data analysis and data application scenarios,a data cleaning framework suitable for perceptual data is designed,including cleaning requirements,data collection,cleaning data,and accuracy evaluation.One of the most important causes,data cleaning methods,and specific process design.According to the characteristics of perceptual data,a method for cleaning four types of dirty data is designed,and the data cleaning process in order of redundant cleaning,conflict cleaning,anomaly cleaning,and missing cleaning is designed from the results of the cleaning.Following the point mentioned above,the design of abnormal cleaning methods.According to the characteristics of the normal distribution in probability theory,the small probability data with a deviation greater than twice ? in each set of perception data is judged,and it is cleaned by deleting them.Another important point,data spatial and temporal correlation design.According to the time series characteristics of the perception data,a sliding time model is established;according to the two spatial characteristics of the perception data,a spatial model of multi-sites and adjacent nodes is established.The ST-SDC algorithm is designed based on the time and space model,and the space-time correlation model is established by the weighted average method.Finally,the design of cleaning evaluation methods.According to the spatial and temporal correlation model established in this paper,the last missing data in the cleaning process is filled.Comparing and calculating the commonly used time ARMA model,spatial VAR model and separate temporal or spatial model to fill the RMSE accuracy value of the data set.In order to complete the research of perceptual data cleaning based on spatiotemporal correlation,first of all,this paper describes the research background and current status of spatiotemporal correlation data cleaning in the Internet of Things,and then introduces the theory of data collection,spatiotemporal correlation and data cleaning,next designs the above five The core content of the aspect,finally based on the analysis of the experimental results and the insights in the research process,summarized the core content and innovation of the research results,and summarized the deficiencies and defects in the research,based on the deficiencies for the future research prospects.