Researches On Sensor Data Storage Strategy For The E-Science Platform Of Forest Ecological Monitoring In The Western Tianshan Mountains

The forest resources in our country are abundant and various, and the different monitoring stations distributed throughout the country and equipped with all kinds of sensors have been gathering massive data. Mass file needs a more effective strategy for storage and management. It was found that there were more or less some abnormal data in the originally collected data by the sensors. Therefore, in this paper the abnormal value has been analyzed and studied, and an error elimination algorithm for the original data was developed. Currently many errors and abnormal value were existed in the original data gathered by all kinds of sensors, some of them were caused by the extreme weather conditions, and others by sensor fault, outside interference, and etc. So manual handling exceptions of data must be done before storing. The accuracy of data was also affected by man-made factors. In order to solve the problem, the negative air ion concentration data gathered from the western Tianshan forest ecological monitoring system were used as experimental data to design a set of automatically error eliminating fitting algorithm based on multiple regression, which could simplify data processing program and reduce the impacts of the existing error of the sensor monitoring original data and the man-made factors. The fitting of this algorithm has an accuracy rate of 86%.In this paper, relational databases and non-relational databases were compared with advantages and disadvantages in dealing with massive data, of which 8 main NoSQL databases were analyzed with the characteristics and application scenarios. Then combining the analysis result with the characteristics of sensor data files in cloud platform 3 candidate databases were selected. Finally, by quoting previous performance testing results of the 3 databases and considering sensor data storage requirements of cloud platform, the HBase database was chosen as a file storage platform for the massive sensor files.In order to effectively exploit HBase database's advantages and improve the sensor data storage efficiency, different storage methods were designed to adapt data with different types and characteristics. Different kinds of sensor original data were separated through a file distribution middleware and then stored into HBase database respectively by its corresponding method. Remote sensing images and video monitoring data, which have bigger sizes, were cut by database into several equally sized storing data blocks for storage while picture type monitoring data, which have smaller sizes, were merged into a larger storing data block which was the same size as the former, and the selected text data were firstly processed using automatic error elimination algorithm and then stored into the HBase database.The prototype system in this paper was the western Tianshan forest ecosystem e-Science platform, and the storage strategy designed for massive sensor monitoring data was applied. A commonly-used storage strategy was also used for a comparing test, which was to analyze the designed method's applicability and feasibility. It was showed that the designed storage strategy for massive sensor data could significantly improve the efficiency of storage and reading, especially for text data and image data. So the designed storage strategy was of practical value and actual significance.