Research On Hierarchical Storage And Query Method For Iot Big Data

Nowadays,Internet of Things(IoT)has become inseparable from our work and life.With widespread application of various sensory technologies and rapid development of IoT industry,sensory equipment for IoT is increasing exponentially in terms of its quanity and various sensory equipment collects information dynamically and sends back data in a real time manner.Thus,massive sensory data forms IoT sensory big data.How to conduct efficient storage,query and analysis of IoT sensory big data has become a key issue to be solved.In addition to conventional big data characteristics,IoT sensory big data also has such characteristics as frequent update,temporal-spatial correlation,flow accumulation,continuous sequence and multi-dimensional characteristic,thus poising a great challenge to conventional big data technology.Based on data layering idea,using existing big data technology and theory,this article implements the IOT-HSQM(IoTHierarchical Storage and Query Management platform which offers a solution for nearreal-time storage and quick query of IoT sensory big data.Its main contributions are as follows:1.This article designs and implements the IOT-HSQM system model which,based on the divide-and-conquer idea,divides the storage of query of sensory data into micro sensory data layer and meso sensory data layer.The former mainly involves storage optimization and query optimization of raw sensory data and post-cleaning effective data;meso sensory data,as aggregation and statistics of micro sensory data,mainly involves storage optimization and query optimization of meso sensory data.2.This article designs and implements the storage and query system for micro sensory data layer,and proposes a time-space block preprocessing storage method which significantly raises the speed of near-real-time storage and write of micro sensory data through such technologies as time-space block preprocessing,data compression,cache batch write.3.This article designs and implements the storage and query system for meso sensory data layer,and proposes a history-based cache replacement query methodHRPB(History-Related Priority-Based Strategy method)which improves cache hit rate and query performance by identifying stage-based hotspot data more effectively.With real history taxi data of Beijing in 2012 and simulated data based on real data expansion,this article demonstrates through experiments that TSBPS has significantly improve the write speed compared with conventional methods with maximum write speed improved by 40% and average write speed improved by 20%;HRPB query method for meso sensory data layer has greater advantage than the LRU-based query method in identifying stage-based hotspot data with cache hit rate improved by 30% upon optimal condition;query speed of meso sensory data layer in the IOT-HSQM system model is much higher than that of conventional query mode.Based on above,the IOT-HSQM system model has evident advantages than conventional big data processing method with respect to near-real-time storage and quick query of IoT sensory big data.This article has positive implication and practical value for promoting the development of IoT sensory big data technology.