Research On The Application Of Kafka-based Message Queue In The Processing Of Astronomical Data

The amount of data generated by.MUSER(Mingantu Ultrawide Spectral Radioheliograph)every day is up to terabyte,and MUSER has to implement the real-time and offline processing of massive data.In the research and development of MUSER,how to solve the problems of massive radio astronomy data processing well is an important issue.In order to meet the requirements of real-time and parallelism for massive data processing,massive data processing systems often use message queues to improve the overall data processing ability.Based on the research of massive radio astronomy data processing in MUSER,this paper does an analysis of the massive data processing requirements in MUSER and makes some insights into message queues.In order to improve the overall data processing ability of MUSER,the main work of this paper is as follows:(1)After the technology selection of message queues,this paper chooses to apply Kafka in MUSER and divides massive data processing into two parts:task publishing and task processing,which can improve the parallelism and real-time of MUSER.In addition,this paper implements the multitasking parallel processing of MUSER in order to improve the overall data processing ability of system preliminarily.(2)MUSER needs to realize the load balancing of consumers to optimize the task scheduling.But the native load balancing algorithm for Kafka consumers can't meet the requirements of MUSER.This paper puts forward an optimized load balancing algorithm for consumers.In order to improve the overall data processing capabilities of MUSER further,this algorithm optimizes the process of load balancing by choosing a manager to control the load balancing of consumers.And this algorithm improves the fault tolerance of MUSER by choosing a monitor to monitor the exceptions.(3)When MUSER uses Kafka to process massive data,the consuming exception would reduce the accuracy of radio astronomy data processing results.This paper designs and implements the consuming exception handling mechanism.In order to improve the reliability and fault tolerance of data processing in MUSER,this mechanism carries out the double-ended log management,exception handling and consistency checking of offsets.The accuracy of massive radio astronomy data processing would be improved by this means.The study results of this paper improve the overall data processing ability of system and enhance the reliability and fault tolerance of MUSER.The study results of this paper have been applied in developing software packages of astronomical massive data processing and achieve good results.It can also provide the reference for developing other massive data processing systems and software.