Research On The Key Technology Of Massive Time And Frequency Scientific Data Management And Service

With the increasing dependence of national defense,scientific research,economic and social industries on high-precision timing service,in order to improve China's high-precision time synchronization service level and support capabilities,the country has successively developed high-precision ground-based timing systems,national time frequency systems,The construction of service-oriented basic scientific equipment and test instruments such as the space station time-frequency experiment system will incorporate time-frequency services into infrastructure construction,and time-frequency scientific data will enter the era of big data and information services.Efficient management of massive time-frequency scientific data is not only the basis for realizing the storage and analysis of time-frequency scientific big data,providing efficient time-frequency information services,but also providing technical support for knowledge acquisition and scientific discovery of time-frequency scientific big data.This paper studies the management system,storage and parallel processing methods of massive time and frequency scientific data from the perspective of service-oriented,and uses cloud computing and big data processing technology to study the key technologies of service-oriented scientific data management of massive time and frequency.(1)Researching and designing a service-oriented integrated management model,technical architecture,and diversified service methods for multiple and massive time-frequency scientific data: aiming at the various differences in management specifications and data standards for independently-built time-frequency scientific data management systems,Causes complex data management,difficult data use,and single data product issues.Based on the characteristics of time-frequency scientific big data and service-oriented application requirements,a data management standard system and prototype system were constructed.Adopting the design concept and loosely coupled characteristics of service-oriented architecture,and comprehensively using a variety of data storage management and analysis methods,a service-oriented multi-mass time and frequency scientific data integrated management model,technical architecture and diversified service methods are designed.The key technologies have been improved to achieve efficient management of massive time-frequency scientific data.(2)The unstructured,semi-structured,and time-series data of massive time-frequency scientific data are studied,and multiple data structures coexist with data storage management methods.Aiming at the characteristics of massive time-frequency scientific data with multiple data structures,combined with the service-oriented mass history Application scenarios for fast data query and online data real-time reading and writing.The combined storage management and table design method based on "distributed file system + distributed column database + time series database" is researched to solve the traditional centralized time-frequency scientific data,low unified storage efficiency,and complex expansion problems.At the same time,based on the characteristics of time-frequency scientific data with many files,small amount of data,and association analysis,a multi-copy hash time-frequency data distribution algorithm that uses small file aggregation and considers data correlation is proposed,which further improves the time-frequency scientific data storage management efficiency And storage resource utilization.(3)The offline parallel analysis method of massive historical data under the framework of distributed parallel programming is studied.In view of the large amount of data and calculation in the offline analysis of massive historical time frequency scientific data,the traditional time-frequency analysis mode has low calculation efficiency or cannot be calculated.A distributed parallel analysis method based on the Map Reduce parallel programming framework is presented.According to the multi-order sliding difference characteristics of the time-frequency analysis algorithm,the time-frequency scientific data segmentation and matrix block method are proposed.The time-frequency analysis algorithm is optimized while the time interval error and the frequency source clock difference model parameters are calculated in parallel.Test the performance of the calculation method.The results show that the parallel massive offline time-frequency analysis method proposed in this paper can effectively solve the problem that traditional single machine cannot effectively calculate when the amount of data is large and the calculation is complicated.(4)Aiming at the problem of massive time-frequency scientific data analysis with frequent and high real-time interaction of service-oriented applications,a method for real-time analysis of massive time-frequency scientific data based on Spark memory parallel computing framework is studied.Realization of Allan and MTIE calculation methods based on Spark parallelization.At the same time,in view of the typical measurement data removal error in time-frequency analysis,the research of ODTD based on optical fiber time comparison was carried out and Spark-ODTD was realized.Experiments verify the effectiveness and performance of the algorithm.(5)For the needs of major scientific projects such as high-precision ground-based timing systems,national time-frequency systems,and time-frequency users for remote high-precision time comparison,synchronization,and traceability services.The data management system and related technologies and methods proposed in this paper are used to optimize the system architecture,data storage and processing methods,and service modes of traditional satellite common-view remote time comparison systems.The cloud service prototype system designed and implemented can support the online processing and interaction of business data of 10,000 terminal devices and online access of 1,000 users through dynamic expansion.The research in this thesis will further improve the standardization management of time-frequency scientific data,and provide standardized data and technical support for the scientific development and application of service-oriented time-frequency time-frequency.It solves the problems of low efficiency and large amount of time and frequency scientific data storage management,difficult data analysis,or uncalculation.At the same time,the massive data storage and parallel analysis techniques in this paper provide a solution for the design of the distributed time-frequency data center architecture for major scientific projects such as the national time-frequency system and high-precision ground-based timing systems,and for the acquisition of time-frequency scientific big data knowledge and scientific discovery.Laying the foundation.