Study On Big Data Storage And Development Of Data Service System For CSES

The Zhangheng-1 satellite,also known as CSES,is a low-orbit satellite launched by China for the specific purpose of electromagnetic observation of earthquakes.Some dozens of GBs of the spatial physics field data is generated every day,which are stored in the file format of HDF5 and transmitted by FTP protocol.However,this storage mode is subject to various limitations such as poor access efficiency,inadequate security measures and imprecise query capabilities which severely impede the production process.This article delves into the HDF5 file storage format for satellite data.It is characterized by segmented storage based on time series which each segment of data has a fixed number of sampling points and lasts no longer than 2 seconds.Based on extensive literature review,we propose a kind of big data storage solution for CSES that combines HBase with Elasticsearch using the Hadoop framework.This approach fully leverages the high reliability and scalability features of distributed processing software architecture as well as the indexing capabilities of HBase Row Keys.The satellite data and index fields are stored separately in HBase tables and Elasticsearch index tables respectively,which achieves secure storage,precise queries and rapid access for massive amounts of data.According to the ideas of software engineering,which involves following steps such as requirement analysis,functional design,database system selection,development framework selection,environment selection,debugging and deployment,implementation and testing,etc,we have developed a set of data service system for CSES.This software system achieves precise data query function for any time period and any region range,with the characteristics of user-friendly interface,practical function,safety and reliability,and stable performance,which greatly facilitates users.The main innovation points of this paper are as follows:(1)The characteristics and models of big data storage technology have been analyzed,and the HBase big data storage solution has been proposed and applied to the CSES data service system,greatly improving the efficiency of data storage and access.(2)According to the requirements for storage and reading of space electric field data on the CSES,the advantages and disadvantages of the HBase big data storage scheme and the Elasticsearch search engine have been analyzed.A big data storage scheme based on the combination of Elasticsearch and HBase has been proposed.Experiments have shown that this storage scheme has higher access efficiency.(3)A satellite data service system based on HBase and Elasticsearch big data storage solutions has been developed,which enables precise query and secure download of satellite data through multi-dimensional condition combinations,such as data query in any time period,any location range,any orbit number,etc.,greatly facilitating user applications.