Research On Lightweight Scope Indexing Of Monitoring Data

In the process of using sensors to monitor the environment, recording the state of the object will produce a lot of data, which is widely used in scientific research. With the enhancement of the data acquisition rate make the existing relational database does not meet the requirements of write-intensive tasks, and cannot be as flexible as non-relational databases to effectively handle large unstructured data. HBase as a key-value storage system can handle this massive, unstructured data well, and to handle the write-intensive tasks. HBase provides the primary key to query the data in a manner to meet the requirements of the primary key query, but for non-primary key queries to spend a lot of time for a full table scan. To solve the problem, we studied the data of non-primary key index in HBase, the index data storage and abnormal warning, that design and realization of wisdom barn cloud services platform, and verified the actual system.For HBase key index for non-primary key query inefficiencies, this paper proposes a method for non-primary key index to retrieve the stored observation data in HBase, which takes advantage of the characteristics of the observed data continuity, we designed a new range of indexing algorithm to solve the problem of non-primary key query inefficiencies.For security and integrity of the index file, we use HDFS to store the index file to ensure the security and integrity of the index files. While other users can easily use the index to improve the utilization rate of the index.For conventional recording level index, such as B + tree, LSM tree, when the index update will have a serious index maintenance costs, which cannot meet the sensor data such writeintensive tasks. This paper proposes a packet indexing method which is a lightweight index structure, and the index files is small compared to the conventional recording level index, that has a lower maintenance costs and improve the speed of index creation.For the problem of abnormal and unusual for sensing data of the object, this paper propose a policy that compare the same physical variables monitoring of real-time sensor data with the adjacent sensor and compare monitoring physical variables with the predetermined threshold value. When monitoring data is abnormal, an alarm condition information established in accordance with the feedback.Through theoretical analysis and experimental verification, the use of grouping strategy proposed the creation of a range of methods compared to a single index using key search methods greatly improve the retrieval performance, improve the efficiency of creating an index, to meet the high write throughput this observation data requirements. The use of HDFS to store the index files to ensure the security and integrity of the index file, improve the utilization of the index.