Design And Implementation Of Natural Resources Big Data Storage Platform Based On Hybrid Architecture

Natural resources refer to the natural elements with economic value or social value.Over the years,China has accumulated a large amount of data in the field of natural resources,such as land,geology and mineral resources.However,due to the limitation of data processing capacity and the lack of unified data standards,although these data are stored and managed in the form of spatial database and relational database,the data are too much and difficult to use,and it is difficult to meet the needs of unified management of natural resource data.The storage and retrieval of natural resources data is always a difficult problem in the research of natural resources data storage and the data application of relevant management departments.In this dissertation,the natural resources data in different sources,different types,management difficulties and other issues,proposed a hybrid architecture to store natural resources data,natural resources big data storage as the core,research and design of a mixed-architecture-oriented distributed storage platform,using distributed storage technology to solve the multi-source heterogeneous mass natural resources data hybrid storage and call,the main research work of the paper is as follows:(1)Using Ceph distributed storage architecture and Mongo DB database to realize remote sensing tile data storage,based on the tile pyramid storage of Ceph distributed storage architecture,combined with TMOSM optimized storage method to achieve rapid access to remote sensing tile data.Simplifies the storage and writing of vector data using Mongo DB’s document database schema.(2)I In order to improve the efficiency of data storage,this paper proposes a new vector feature storage framework,which uses Geo JSON format and Mongo DB database cluster replication technology,so as to effectively solve the storage problem of massive vector data.The construction and some functions of natural resources big data storage platform are realized.Hadoop cluster and Ceph cluster are built,and functional and performance tests are carried out using natural resource image data.The results show that the data writing performance of Ceph distributed storage architecture is slightly better than that of HDFS distributed storage architecture,and the performance advantage of Ceph distributed storage architecture is more obvious when storing massive natural resource data.