The Space Science Big Data Storage Model-SP-HDF And Applied Research

Space science is the study of natural phenomena of physics, astronomy, chemistry andlife and it’s laws which occurred in terrestrial space, interplanetary space until the entireuniverse. Space science observations and research processes will generate a huge amount ofvarious scientific datas which are complex and compute-intensive. Effectively storing,managing and sharing these scientific datas has become an urgent issue for applied researchof space science. Based on the analysis of the characteristics of massive space science datas,this paper studies the HDF(Hierarchy Data Format) and CDF(Common Data Form) storagetechnology in-depth. On this basis, we propose a self-described space science big datastorage model-SP-HDF. Considering the coexistence of standardized and non-standardizeddatas, the authors propose a shared services model based on virtual observatory. The researchresults and innovations include:(1) The author propose a self-described space science big data storage, management,integration and sharing model-SP-HDF. According to the characteristics and logicalrelationships of the science datas from space weather and Meridian Project, we designedtheir logical storage structures. On this basis, the author designed a friendly data accessinginterface which is simple and easy to use. The SP-HDF storage model has the advantage ofshielding the details of directly accessing the underlying HDF and CDF files, therebyreducing the complexity of data accessing. The other hand, the logical storage structure bereleated with application fields directly, which improved the abstraction level of dataaccessing and data granularity. It is more understandable.(2) According to the circumstance that the massive science datas can not be completelystandardlized in a short period of time, we proposed a shared services model based on virtualobservatory. The model takes advantage of shared metadata model and abstract data serviceinterface to achieve data integration and shared services for diverse, heterogeneous scientificdata.(3) We have designed and developed an interface prototype system involving the logicstructures and the service interfaces for big science datas of space weather based on theSP-HDF storage model. Taking advantage of these core data accessing interfaces, we havecomplished the task of several integrated applications with the numerical simulation systemsand visualization systems easily. We verified the data service interface prototype forMeridian Project based on virtual observatory, and implemented data retrieval, datadownload, data visualization and many other features. (4) The author have designed the standardlization model for Meridian Project based onSemi-formal descriptive XML and data processing template technology. We have overcomedthe difficulties for efficiently data processing and flexible customization of diverse,heterogeneous, massive space science datas.