Research And Implementation Of Massive Data Processing And Storage Architecture Of Beidou Monitoring Receiver

The Beidou navigation satellite system is a global navigation satellite system developed by China itself,which has great strategic significance for China national defense security and national livelihood.The monitoring receiver analyzes the satellite signal integrity through fixed protocols and standards.Only when the monitoring receiver is in the normal working mode,the correct analysis result can be obtained when the satellite signal is evaluated.Therefore,the test evaluation of the monitoring receiver is the most important production link for each monitoring receiver manufacturer.The traditional monitoring and evaluation of monitoring receivers is often done manually.The real-time streaming data of the receiver is received and stored by means of signal analysis software,and then offline testing is performed by means of evaluation methods one by one and one by one.In the case of longterm reception,huge amount of data will be generated.Data collection takes a lot of time and manpower,and related auxiliary software is outdated,human participation is large,realtime results cannot be given,and performance is extremely unstable.Test evaluation requirements for the fourth generation navigation system.Therefore,researching and implementing a processing and storage architecture that supports long-term data processing,supports real-time data evaluation,and can support multiple monitoring receiver test evaluations at the same time will greatly improve the efficiency of monitoring receiver test evaluation,and smoothly carry out satellite navigation work.There's important meaning.First of all,this thesis designs the overall test evaluation process of the monitoring receiver,and designs the data receiving and data processing and storage process in detail.On this basis,this thesis implements the C++ serialization test evaluation code under a single machine,and uses Visual Studio to introduce the bottleneck analysis of the implementation code.Through bottleneck analysis,it was found that there was a problem of insufficient single-machine memory in the serial test evaluation process,which took a long time and could not be evaluated in real time.At the same time,according to the test data arranged in time series,each test item does not interfere with each other,and each frequency point and the test evaluation under the satellite can be separately performed,which meets the requirements of parallel processing.In response to these problems,a scheme for using parallelized reception and processing is proposed.Different receiver data is mapped to different computer nodes through a distributed consistency hash algorithm,using Kafka message middleware to buffer data and decouple the data receiving and data processing modules,and using Flink to parallelize multiple test items,Receiver data,and finally store the test evaluation results and the received data in the HDFS distributed file system.Secondly,the data processing and storage architecture of the monitoring receiver based on Flink + Kafka + HDFS proposed in this thesis is implemented.The implementation process is divided into three parts.The first part is the data reception and Kafka real-time data transmission implementation.According to the receiver data characteristics,the data serialization and deserializer are implemented for each test item data.The second part is based on the four typical data processing methods summarized in the bottleneck analysis.The bottleneck calculation process of the monitoring receiver uses Flink parallelization processing.The third part is the storage implementation of the receiver evaluation result file and the receiver data in the HDFS cluster.Different storage paths are created in HDFS according to the test items and the frequency point number,and the result file in this thesis is characterized by small files.The original storage algorithm is optimized,and the HDFS storage structure is optimized by combining the small files into a combined file and adding an index table mechanism.Finally,the experiment and analysis of the data processing and storage architecture of the monitoring receiver based on Flink + Kafka + HDFS proposed in this thesis,including functional test and performance test.The experimental environment is a distributed system built by four PCs.Functional tests show that the distributed architecture proposed in this thesis can use TCP to receive and monitor the receiver signal and parse it,and transmit it to Flink in real time through the message queue Kafka.Parallel processing of receiver multiple test item data and multiple monitoring receiver data in Flink.And in the data processing process,the original data and test results are stored in HDFS.Through the parallelization of the processing flow,the distributed architecture data processing proposed in this thesis has an average speedup of about 4 compared with the original serialization algorithm,and the receiver test evaluation performance has been significantly improved.The distributed processing and storage architecture based on Flink + Kafka + HDFS implemented in this thesis is applicable to the batch test evaluation process of monitoring receivers.