Design And Implementation Of MUSER Data Archiving And Publishing System Based On NoSQL Technology

With the high spatial, high temporal and high frequency resolution observation equipment put into use, astronomical observation data rises rapidly. Traditional scientific data storage and processing method is far from enough to satisfy the needs of numerous astronomical data. In order to improve the efficiency of the scientific output of modern astronomical observation, there must to explore new methods of data storage and archiving.Based on Mingantu Ultrawide SpEctral Radioheliograph project as the research background, as the project enters the observation stage MUSER, this article studies how to quickly and efficiently store its multi-band multi-channel observation generate massive amounts of data. At the same time, for convenience of observation data sharing, we should take effective ways to release observation information on the basis of the archive. This research work mainly includes four aspects.The first, on the experiment contrast through FastBit and the traditional relational database named MySQL based on the technology of NoSQL in indexing, single (multi-) field queries, inserting data and storage space, draw FastBit is more suitable for MUSER actual indexing and retrieval requirements.Second, it designed and implemented data archiving systems of the MUSER project, including data archiving of FITS,UV-FITS format and PNG format. FITS format data archive and at the same time index for retrieving simply. PNG format data applies in the project of real-time release, on the basis of the characteristics of MUSER according to the days of observation, in real time to create archive directory hierarchy by day.Third, using related technologies to combine the images through MUSER and and related information for MUSER release system.Finally, MUSER project data publishing system was designed and implemented. The release includes every 10-minute images and daily transit time images of dynamic publishing. At the same time, it makes the image of every 10-minute into animation effects to day, and the image of transit time into animation effects for release in month. Animation has accelerated playback, slow play, start playing, pause playback and other effects.In this article, the research and experimental data archiving and distribution system has been successfully applied to the current MUSER data processing and analysis system.