Research On Key Technologies Of Geospatial Real-Time Stream Data Compression And Query

With the development of temporal geographic information systems and the popularity of smart mobile devices,massive geospatial data is being generated all the time in the world.A large amount of valuable knowledge and potential regular information is stored in the data,and the value of these data will decrease when time flies.Therefore,it is particularly important to process geospatial data in real time.The traditional processing method of geospatial data is loaded and stored in the file or database into the memory at one time,and processes the data to feed back the results to the user.This method has many drawbacks.Firstly,the stream data is generated continuously and has infinite characteristics.The traditional analysis method needs to load all the data into the memory and perform subsequent analysis.When the data volume is too large and the memory is limited,the traditional analysis method will be invalid.Secondly,the traditional processing method usually processes the data cyclically when processing the stream data.When processing the stream data,the analysis method can usually only scan one time.Each data only will be processed once.In addition,the traditional processing method is independent in processing the stream data each time,which results in a large number of meaningless operations and reduces processing efficiency.In view of the above problems,this paper designs geospatial real-time streaming data compression and query technology based on core technologies such as Spark Streaming and HBase.The technology realizes real-time compression,storage management and continuous querying of geospatial data.It has the advantages of high efficiency compression and real-time query.The main work of this paper is as follows:(1)According to the real-time characteristics of geospatial real-time streaming data,the CE trajectory compression algorithm is improved.Combined with time attribute and spatial attribute information as threshold,T-CE trajectory compression algorithm was designed.The algorithm avoids that the representative points in the curve will be deleted by mistake.(2)According to the structural characteristics of geospatial data and the storage characteristics of HBase,the storage model of geospatial data in HBase was realized.Different row keys were designed according to different storage requirements during trajectory compression and continuous query.(3)According to the characteristics of dynamic update of track point data,a concept partitioning method based on quadtree index was proposed.By accessing data in this way,it can better adapt to the real-time updating characteristics of track points and solve the problem of simple mesh index due to data.Uneven distribution causes data imbalance loading in each grid.(4)Based on the concept partitioning method of quadtree index,continuous k-neighbor query and continuous range query were realized.The historical query result is updated based on the change of trajectory data in the query process to obtain the latest query result.Through the result update method,a large number of repeated calculations of the query work are avoided,and the query performance is improved.Finally,this paper builds a cluster environment and conducts related performance tests based on T-Drive Beijing taxi trajectory data.The experimental results show that the geospatial real-time streaming data compression and query technology studied in this paper is superior to similar schemes in terms of continuous query performance.In terms of real-time compression,the proposed T-CE algorithm has the characteristics of fast compression and small error.