| The Square Kilometre Array(SKA),the world’s largest radio telescope,is nearing completion of its first phase(SKA-P1),and scientists will use the SKA to conduct large surveys including continuous spectra,spectral lines,and transients sources to investigate key scientific questions.The level of processing and research of scientific data from the survey will directly affect the output of scientific results,so it is urgent and critical to study efficient data processing methods.The significant improvements of SKA-P1 in dynamic range,field of view,resolution,and sensitivity result in a tremendous increase in the amount of data produced by SKA observations,estimated to be around 710 petabytes per year.However,existing spectral line imaging and transient source search pipelines have low imaging efficiency,poor imaging quality,low efficiency in distributed processing of large-scale data on supercomputing systems,poor stability,and inadequate scalability,making them unable to meet SKA’s scientific data processing needs.In this paper,the efficiency and quality of key imaging steps in the pipeline,as well as the stability,operational efficiency and intelligence of the entire pipeline,are investigated in three areas:(1)To address the problems of low efficiency,poor scalability and inadequate spectral imaging quality of the existing 21-cm spectral line imaging pipeline for processing SKA spectral line data,an SKA 21-cm spectral line imaging pipeline based on Data Activated Liu Graph Engine(DALiu GE),a data-driven distributed execution framework,is proposed.The WSClean imaging algorithm is used for deep imaging of spectral data and improving imaging efficiency.Parallel optimization is carried out for spectral line imaging and imaging source searching according to the frequency channel of spectral data.The pipeline processing task is developed as a Drop and integrated into the DALiu GE framework,improving pipeline stability,efficiency,and scalability,and providing a feasible solution for processing massive data for more SKA scientific tasks.The development of different processing tasks of the pipeline into multiple Drops integrated into the DALiu GE framework has improved the stability,scalability and about 7% operational efficiency of the pipeline,providing a viable solution for more scientific tasks of SKA to process massive amounts of data.(2)To address the low performance and accuracy of the slow transient source search pipeline for variables and slow transients(VAST)survey in the Australian SKA Pathfinder(ASKAP),this study improves the imaging part of the original pipeline using the W-stacking algorithm of WSClean,performs parallel processing for critical steps,and optimizes the improved VAST slow transient source search pipeline by integrating it into DALiu GE.Experiments are carried out on the hardware platform of the SKA Regional Centre prototype(CSRC-P)in China using data obtained from beam 29 of VAST survey SB9602 observation.The experimental results show that the scalability,stability and flexibility of the pipeline have been improved,while the accuracy of the slow transient source search has been increased by 16%,and the overall operational efficiency of the final improved pipeline has been increased by a factor of about 3,effectively handling large batches of VAST full-scale survey data.(3)To address the problem of the VAST slow transient source search pipeline’s dependence on human eyes to resolve slow transient sources and imprecise shape information of celestial bodies,a slow transient source detection method based on spatiotemporal joint features is proposed,and deep learning is applied to slow transient source detection.High-redundancy celestial bodies are filtered out through repeat rate screening,and potential transient sources are selected through variability screening for detection.A spatiotemporal joint feature extraction network is constructed for slow transient source detection,automatically learning the features of slow transient sources in time and space.Finally,the detected slow transient source images are segmented and classified.This chapter’s method simplifies the detection steps of traditional methods and reduces dependence on human eye resolution,improving detection accuracy and obtaining more accurate shape information for subsequent research.Finally,perform slow transient source detection and classification on the preliminarily screened images to achieve image segmentation.This chapter’s method simplifies the detection steps of traditional methods and reduces dependence on human eye resolution,improving detection accuracy and obtaining more accurate shape information for subsequent research. |
