Research On Key Technologies Of Vector Big Data Management

In the era of big data,data has become an important part of national basic strategic resources.With the rapid development of spatial information technology(GIS?RS?GPS),the collection of spatial data is becoming more and more diversified,and the GIS application in all walks of life is more and more extensive,which leads to the arrival of vector big data age.Big data is a "double-edged sword".The high precision,wide coverage of spatial vector data explosion growth,provide a good opportunity to enhance the national macro-scientific decision-making,social supervision,public services and emergency decision management capabilities.However,how to effectively organize and manage the ultra-massive vector data to maximize the benefits,has become a big problem in the practical application.The sudden emergence of the cloud computing platform Hadoop,which through the distributed storage strategy and parallel computing technology in cluster,can obtain a good system performance,and high scalability advantages to meet the growing demand for data processing capacity,and has become one of the mainstream technology for big data analysis.Based on aboves,this paper explores and studies all aspects of data storage,indexing,query and visualization for vector big data management and application model.A set of key technologies are presented.At the same time,it tests and analyzes the practical application requirements of the national arable land quality management system,which verifies the feasibility,practical significance,and use value of our research contents.The main research contents in paper include the following four aspects:(1)Research on vector big data cloud storage modelIn order to meet requirements of vector data processing and analysis in cloud environment,this paper proposes a vector big data cloud storage model-GeoCSV,which is based on the characteristics of vector data and the advantages of Hadoop cloud platform.Firstly,the data structure of common vector data storage models are discussed.Secondly,the Hadoop cloud environment is studied,and base on analyzing the advantages of object-oriented spatial geometric element model,cloud storage model——GeoCSV is designed and implemented with the Key-Value model.(2)Research on distributed R tree indexing method of vector big dataIn order to improve the retrieval efficiency of vector big data,this paper designs a distributed R index framework based on HDFS.Firstly,based on the principle of spatial indexing and advantages of distributed storage,the distributed index mechanism in cloud environment is discussed.Secondly,based on the characteristics of spatial distribution and different sizes of vector data,the data partitioning strategy based on spatial coding is designed respectively,and the parallel construction of distributed R tree index is realized.Finally,through experiments,the efficiency and feasibility of different distributed spatial indexing algorithms are verified from two aspects:spatial index quality and load balancing.(3)Research on parallel processing of vector big dataBased on above research results,this paper carries out parallel processing methods related research for vector big data.Firstly,the parallel conversion algorithm of vector data is realized based on MapReduce,and the vector data is transformed from the Shapefile to the GeoCSV with the Key-Value model.Secondly,for the large-scale spatial query demand,vector data parallel query algorithm is implemented.Then,aiming at visual browsing and viewing of large-scale vector data,the parallel construction type algorithm of vector data tile pyramid model is designed and realized.Finally,combined with the real datasets,experiments are used to verify the efficiency and feasibility of the proposed vector data processing algorithm.(4)Management application of national arable land quality in cloud environmentBased on the research results of the key technologies for vector big data,this paper designs the framework of the national arable land quality management in cloud environment,which is based on the practical application requirements of land information system.Hardware and software environment were deployed,and the key technical content and application cases were tested and analyzed.Through this section,on the one hand,it is to verify the feasibility of the content in this paper;on the other hand,it also shows that the content of this study has a certain practical significance and useful value.