Research On MUSER Data Processing Pipeline Based On Flow Calculation

Mingantu Ultrawide Spectral Radioheliograph (MUSER) is an solar observation radio interferometer array located in Inner Mongolia. As a dedicated interferometer array of solar, MUSER can form a long time, a large space and high-quality audio spectral resolution image between the frequency range of 400MHz to 15GHz. Ideally, the system should process MUSER data real-timely, including data acquisition, storage, processing, release. MUSER-I can produce nearly 1.05TB of datas a day from ten hours observation, and MUSER-II can produce nearly 2.146TB every day. At the same time, the operating system monitoring the entire telescope needs to generate the image in order to determine the status of the telescope as soon as possible. The output of MUSER data per day is huge, and it urges to establish a Pipeline to deal with these data.In this scenario, designing an automatic data processing system such as pipeline for MUSER is meaningful. The research work in this paper are as follows:first, We do research to the streaming computing platforms and streaming large data; second, We studied real-time data processing methods for MUSER, and a scientific data processing pipeline based on distributed computing streaming technology is designed; third, the distributed data processing pipeline are implemented by the use of open source framework Spark Streaming. The Pipeline can deal with MUSER observation data such as data preprocessing, image reconstruction, deconvolution, and real-time monitoring. We describe the overall system architecture and concrete realization of pipeline, and the distributed data processing pipeline is proved to automatic, robust, scalable and manageable. Besides, the pipeline run in the CPU/GPU hybrid parallel system to generate UVFITS file that can meet the data processing needs of everyday MUSER on its efficiency.In this paper, We implement astronomical real-time data processing pipeline based on Spark Streaming stream computing platform, which can provide technical support for astronomical observations to some extent and has a positive significance to promote the development of astronomical research.