Design And Implementation Of Massive Astronomical Data Management System Based On Multi-tier Architecture

The in-depth development of astronomically observed technology enables astronomers to obtain massively observed data.The GWAC(the Ground-Based Wide-Angle Camera Array)located at the Xinglong Observatory of the National Astronomical Observatory consists of multiple telescopes,each of which in the observed period monitors a wide range of the sky in parallel so that observations effectively meet the observing needs of astronomers.Astronomers' demand for GWAC observed data can be summarized in four points.The first one is real-time acquisition of astronomical observed data;astronomers acquire observed data during the observed period.The second one is real-time processing of astronomically observed data.Astronomers demand real-time data processing of the observed data to discover astronomical phenomena in a timely manner.The third one is astronomically observed data storage.Astronomical data as valuable scientific research resource necessitates the long-term and reliable storage for scientific research work in the future.The last one is astronomically observed data retrieval.Astronomers claim a data retrieval service out of the system for massively observed data,considering that the deficiency of a solution exists in currently astronomical community about multi-lens parallel in acquisition and processing astronomically observed data.About the astronomically observed data,on the one hand,astronomers request that it can be quickly stored,and on the other hand,astronomers require it to be retrieved in a large-scale.In recent years,the management of astronomical data has received extensive attention from the academic community.A sub-topic,"key technology and system for ultra-large-scale relational data management",which was included in the 13th Five-Year National Key Research and Development Program,has raised the issues referring to management of astronomical big data.Based on the project,the system has designed a multi-layer architecture of data management to operate massively astronomical data in conjunction with the GWAC application scenario.From the overall level,the system is divided into layer of real-time data management and layer of offline data management.The layer of real-time data management mainly processes nightly observed data with less time-consuming procedure,including image processing and cross-certification,and back feed astronomers' observations so as to meet their needs in real-time data processing during the observed period.The layer of offline data management mainly further processes the observed data,persists the data,and provides astronomers with retrieval services for historically observed data during the non-observed period.The system conducts performance tests about the observed data through accessing to the GWAC telescope array,which is located at the Xinglong Observatory of the National Astronomical Observatory.The test results have two perspectives.For real-time observed data,the system can complete the real-time processing in 3 seconds of one shot observed data,which is much smaller than the observed period of the lens of 15 seconds,and can meet the astronomers' requirements in real-time acquisition and processing of observed data on basis of GWAC.For historically observed data,the system can satisfy astronomical data storage of terabytes(PB)level and the delay of querying 2.5 million rows of data from 100 billion rows of historical data is about 12 seconds.In general,the system can necessities basic needs of astronomers for data management.