Research On The Optimization Technology For Meteorological Data Archiving And Retrieval

Meteorological science data is an indispensable crucial resource for business application in numerical prediction,and it is also a cornerstone for supporting the rapid development of numerical weather prediction technology,which embodies the enormous scientific,economic and social value.With the advance of time and the improvement of the precision of numerical forecast,Meteorological science data is continuous increasing in variety and quantity,which has become a challenging research topic to effective use and efficient management.European Centre for Medium-Range Weather Forecasts(ECMWF)in the leading position,developed Meteorological Archival and Retrieval System(MARS),which is currently widely used,and provided massive meteorological data management services for observational data and numerical forecast product data,such as archiving,inquiry.In this paper,we collectively analyzed and researched for the key technology of MARS,and provided corresponding parallel optimization design and implementation for critical performance bottlenecks of query access.The main work is as follows:1.Researching and analyzing the feature for meteorological data structure of GRIB and BUFR encoding type,on this basis,and targeted to carry out the technical development and the analysis of functional characteristics for domestic and foreign meteorological data management platform,and focuses on the key technology of metadata management in MARS framework,which is indexing technique of data cube based on multi-dimensional temporal data structures.2.Carrying out the research on optimization for region query based on retrieval in MARS,and proposing an efficient methods through complement transform range query(CTRQ).For the efficiency bottleneck problem of the existing MARS to execute a wide range of regional query,we designed an efficient methods through complement transform range query based on mathematical complement,and achieved parallel operation for CTRQ algorithm based on high performance computing platform by Open MP hybrid model.This method can effectively reduce the amount of data cube aggregation computation,and further improve the computational efficiency by parallel algorithm.3.Carrying out the research on optimization algorithm for the physical structure reorganized based on archiving in MARS,and proposing an efficient methods of index-mapping parallel processing(IMPP).For the low efficiency problem of the existing MARS to execute updating when mass meteorological data archived,we proposed an efficient methods of index-mapping parallel processing by using multi-thread technology to parallel update.The algorithm can effectively reduce the updated response time of metadata reorganized in physical structure,and achieve the purposes of meteorological science data archived rapidly.