| With the development of remote sensing technology, remote sensing observation data types and levels enrich and show the multi-format, multi-standard, multi-type, multi-scale, mass, and distribution storing characteristics. Many departments and institutions have established remote sensing data libraries which form a distributed, heterogeneous, cross-sector and cross-boundary remote sensing data library group, which has greatly restricted the inter-departmental information sharing and application. In order to meet the needs of the management and sharing of remote sensing data, we implement a large-scale remote sensing data storage systems based on MongoDB and HDFS, this paper describes the detailed design and implementation of the system, and focuses on two key technologies:heterogeneous metadata integration and efficient storage of massive remote sensing data.Owing to the multi-source, heterogeneous, massive features of remote sensing metadata, this paper proposes a mapping template-based heterogeneous remote sensing metadata integration method which can format heterogeneous remote sensing metadata and efficient storage, and provide dynamic expansion ability to resolve emerging new type of remote sensing data.The characteristics of the remote sensing metadata are read-only, small files, massive. The remote sensing image data is not only read-only and massive, but also a single remote sensing image data is large file mostly GB and cold data. The data storage layer use separate strategy and optimization for metadata and image. We use MongoDB-based storage architecture for metadata, which will not only provide efficient data storage, but also has the characteristics of high reliability, high scalability. For remote sensing image we use optimized HDFS with multi-level storage policy in order to improve storage efficiency. The optimization is based on data access heat and combines erasure code and multi-copy, which significant saves storage resources,... |
PDF Full Text Request