Research And Implementation Of High-concurrency Data Processing Of Backup System And High Availability Of System

With the development of China's Beidou satellite construction industry,the government have organized the relevant departments to develop the Beidou Ground-based Augmentation System,and successively built national framework network base stations to communicate directly with satellites and receive raw data.As an enhanced backup center of a data management system which is the sub-system of the Beidou Ground-based Augmentation System,the backup system,synchronized with the data management system,also receives the raw data from the frame network base stations and then performs the standardized processing.As the number of base stations keeps increasing and the size of the real-time stream data is also larger and larger,the original single-machine multi-threaded processing scheme cannot meet the real-time processing requirements of the raw data stream from the base stations.Therefore,it is necessary to research and implement a high-concurrency data processing scheme for real-time data stream from the base stations in the backup system.In addition,when the data management system is unavailable,in the case of transparent to users,the backup system needs to quickly replace data management system to provide services to minimize the impact of single point of failure caused by the data management system.Therefore,from the perspective of providing external services,it is necessary to research and implement a high availability scheme for the overall system.In this paper,a high-concurrency data processing scheme is realized by means of distributed stream data processing and computing framework.The raw data from the framework network base stations is received by using the Flume framework and is preprocessed by packet parsing and time conversion.Then Kafka,a distributed message system based on Zoo Keeper coordination,will receive the data from the Flume framework,and the data is buffered in the Kafka to ensure that the entire data transmission and processing process are smoothly connected.The Flink stream processing framework consumes data from Kafka,and the raw data is converted from binary data stream to RTCM3.2 standard format data according to a given data format conversion algorithm,and then the standardized processing of the raw data from the frame network base stations will be completed.This paper adopts the remote data synchronization scheme based on the Rsync algorithm,which is a remote file synchronization algorithm,and inotify,which is a file system monitoring mechanism,to realize the consistency of the business data and the device status information between the backup system and the data management system.On this basis,when the data management system is monitored to be unavailable,the backup system quickly replace data management system to provide uninterrupted or short-interrupted services by switching the service system in the case of transparent to users.Finally,with the testing and verification of the high-concurrency data processing scheme and the high availability scheme,the experimental results illustrate the validity and rationality of the schemes from our research work.The high-concurrency data processing scheme can achieve low-latency reception and high-concurrent processing of the raw data from base stations,and the advantage of low-latency achieved by distributed parallel computing is more obvious as the base stations' raw data volume increases;The high availability scheme can also achieve that the backup system can quickly take over the service when the data management system is unavailable,from the perspective of providing external services,the high availability scheme for the overall system will be implemented.