Research On Fast Time Series Reconstruction Of Massive Astronomical Catalog Data Based On MongoDB

In recent years,with the improvement of the data collection capability of astronomical observation equipment,the data of astronomical catalogues have shown explosive growth,and astronomy has entered the era of big data,as a result,the efficiency of generating astronomical time series data by traditional scientific calculation methods is not high,which directly affects the scientific output of time domain astronomy,accelerating time series data generation has become an important research direction at present.The traditional relational database is relatively weak in storing and processing massive catalog data.The application of No SQL database in distributed environment provides a new research perspective.In order to solve this problem,this paper presents a fast cross-match method of the same source catalogs based on fast mapping and a MONGODB based application scheme,in order to solve the efficiency problem of batch time-series reconstruction of large-scale astronomical catalogues,we focus on the optimization of original data access,index query and cross-match algorithm.In terms of raw data access and index query optimization,this paper proposes a MongoDB based cross-match algorithm,which utilizes the geospatial index and distributed storage in MONGODB,through the same characteristics of 2D index and HEALPIX index partition in geospatial index,the fast access and extraction of astronomical data can be realized,and the distributed storage of astronomical data can reduce the storage pressure of the master node,MongoDB’s replica set guarantees data security and facilitates distributed and parallel computation of data.In order to solve the problem of huge computation load and low efficiency of traditional cross-match algorithm,a fast cross-match algorithm is proposed in this paper.Since the object studied in this paper is a catalogue of the same origin,it has the feature that the position of the object does not change much,Therefore,this algorithm first determines the areas that need to be certified as special areas and non-special areas through the comparison of quantity and brightness.For the stars in the non-special area,the distance-free matching output is performed by position comparison,and the distance calculate is reserved for the stars in the special area.This algorithm greatly reduces the computational complexity of cross-match,and effectively improves the computational efficiency of large-scale star catalog data cross-match.The experimental results show that this method can improve the efficiency of time series data generation more effectively than the traditional multi-band cross-match algorithm and relational database method,it provides a new idea for the reconstruction of time series and the generation of light curves for the large-scale Catalogue Data of the time-domain Astronomical time-frequency sampling telescope.