Research On GNSS Data Storage And Retrieval Based On HBASE

With the gradual construction and improvement of global satellite navigation systems(GNSS)such as GPS,GLONASS,Galileo and BDS,countries around the world have established their own continuous operation reference station(CORS)system,and applied relevant data and technology to real-time positioning and navigation,Earthquake monitoring and forecasting,geodetic surveying,crustal movement monitoring,urban planning,engineering construction,meteorological services,geographic information systems,precision agriculture and transportation management,etc.The data volume of large-scale observation data and calculation products generated by the GNSS tracking station network has exploded,which brings great challenges to data storage management and rapid retrieval.The file-based storage management method cannot meet the user's need for the flexible use of GNSS data.Although the relational database-based solution can solve the problem of flexibility,when storing and retrieving large amounts of data,there is a performance bottleneck and expansion is difficult to maintain and costly.In response to the above problems,this thesis uses big data technology represented by the Hadoop ecosystem to redefine the storage and retrieval of massive GNSS data,and builds a distributed storage retrieval architecture based on the NoSQL database HBase and the cloud real-time search engine Elasticsearch(ES).Compared with the file-based and relational database-based data management methods,the architecture scheme in this paper facilitates the CORS data center to provide the data and computing environment required for GNSS distributed solutions with higher efficiency,such as large-scale GNSS network baseline solutions,network Adjustment,quality analysis of observation data,etc.The innovations and main research contents of this article are as follows:1)According to the business requirements of the CORS system,five modules were designed,including real-time data storage,batch import of historical data,index creation,data retrieval,and data acquisition,which constituted the core of the storage retrieval system in this thesis.2)According to the characteristics of various types of GNSS data structures,HBase table storage mode is designed to achieve fine-grained management of GNSS data in epochs to meet users' diverse data customization and usage needs.3)Combined with the characteristics of RINEX file format and business requirements,a secondary index structure based on ES is designed.This solution saves the cost of index data storage and improves retrieval efficiency.4)The MapReduce-based GNSS data batch import algorithm is proposed to avoid the low throughput problem caused by calling the HBase client API to import massive historical data.Finally,through the comparative experiment,the function and performance of the system are tested from three aspects: data writing,retrieval and reading.Experimental results show that HBase write and read rates can meet business needs,and ES search response time is maintained at the second level.