Research On Data Management And Data Service Toward Aurora Classification

In the study of solar-terrestrial space physics,aurora is the only geophysical phenomena which can be observed with the naked eye.The systematic observation of aurora can obtain lots of information on the magnetosphere and solar-terrestrial electromagnetic activity.With the development of auroral image acquisition system,aurora is observed continuously and systematically.Massive aurora data is accumulated.To promote the polar scientific research in China,advanced big data management and analysis technologies need to be used.In this paper,the data management and data service toward aurora classification are researched,and the major contributions include:(1)Based on the analysis of aurora classification process,the data in aurora classification is divided into three categories: aurora data,algorithm data and aurora classification data.The concept modeling and logical modeling of data are provided.Moreover,to support in suit analysis of aurora,the service of quick access to raw aurora data is designed based on in-memory database.(2)Based on the lifecycle management for machine learning and deep learning,the process of aurora classification is abstracted and the resampling algorithm based on sliding window is provided.Besides,the data service for the automatic iteration of aurora classification based on Monte Carlo Cross-Validation is designed and implemented.(3)To support the query and visualization of aurora data,the services for metabased query and multiresolution visualization of aurora are designed.The service for content-based query is also designed by using perceptual hashing.Besides,the services for querying “data-algorithm-model performance” data chain are provided.(4)Open source big data processing and analysis frameworks are integrated.The preprocessing of raw aurora data is parallelized by using MapReduce.The content-based query,image feature extraction and aurora classification based on machine learning are parallelized by using Spark.The training and testing of model in deep learning are provided by using Keras.(5)Based on distributed PC clusters,a prototype system for data management and data service toward aurora classification is designed and implemented.