Research On ONSET Distributed Data Processing Framework

The large amount of data produced by astronomical telescope every day puts forward a high requirement for the rapid processing ability of astronomical data processing system.With the rapid development of distributed technology,the concepts of distributed telescope information system,modular design and distributed call and integration have been widely agreed.The existing data processing system of Optical and Near-infrared Solar Eruption Tracer(ONSET)Telescope is composed of several independent parts.Its data processing process relies heavily on manual operation,and the performance of data processing is far from the requirement of its ability to quickly process data.In order to improve the existing data processing methods of ONSET,based on the analysis of the existing data processing process of the ONSET Solar Telescope,this paper studied and implemented the building method of the distributed data processing framework of the ONSET Solar Telescope.At the same time,the data distribution method of image reconstruction module,one of the core modules in the process of data processing,was studied,and some improvements were made,including:?.This paper investigated the intermediate communication technology commonly used in distributed data processing framework,and studied in detail several data distribution methods of high performance communication middleware ZeroMQ.Several application libraries and techniques used in the process of astronomical image data processing were also studied.?.This paper analyzed the existing ONSET data processing programs,including:Image frame selection program,registration and alignment program and image reconstruction program.The communication interface was developed on the basis of these programs by using ZeroMQ communication middleware,and the distributed data processing framework was set up,and the work scheduling plan script was implemented to carry out the overall scheduling of the ONSET data processing process.?.This paper analyzed the existing distributed computing image reconstruction programs based on MPI and CUDA,studied their data distribution methods,and proposed a high performance data distribution method based on a tree MPI CUDA hybrid model.The method was used to simulate the data distribution in the process of image reconstruction,and the effect of data distribution was tested.On the whole,the research and implementation of this framework solve the problem of inefficient caused by manually copying and processing of data in the process of astronomical data processing.And through the research and improvement of the image reconstruction data distribution method,the parallel processing efficiency is improved effectively,and the overall running speed of the whole framework is improved.