Research On Caching Mechanism And Database Optimization Of Massive Meteorological Data

With the development of science and technology in all areas of society, the amount of information from all life walks showing the explosive growth. In meteorology, the meteorological service data and product of the national autonomous regions, municipalities directly under the central government, province, Special Administrative Region and the large or medium-sized city is very impressive, complex, diverse and massive. The timeliness contradiction between meteorological data and service make the complexity and difficulty of meteorological data storage, management and use grow with each passing day in existing meteorological data network. The heterogeneity and geographical distribution of the meteorological data access in trouble, such as the upload of radar base data is every 5 minutes. Most of the time, the interval of 5 minutes cannot meet the data transmission’s network delay between nodes, so the big problem of the storage and access are exposed out. Therefore, this thesis as a starting point, aiming at the problems mentioned above and put forward reasonable scheme of data storage, data access, and to optimize the quality of meteorological service for industrial; performance issues in the process of visiting, will be added to the system database optimization technology. This thesis has completed the following work:We design a meteorological cache framework (MeteCS) based on distributed file system. In this thesis, we study the existing data splitting algorithm which has an alias that is data placement algorithm, and design the hashing data placement algorithm (MeteHash) based on the consistent hash data placement algorithm with considering the performance difference between heterogeneous nodes in a distributed system and loading balancing of nodes.Cache replacement algorithm based on Greedy dual size frequency is proposed potential cache replacement algorithm (WGDSF) which is based on the weight.In WGDSF, we study three parameters that based on the frequency of weighted time, the cost of weighted size, the content of weighted type. The performance comparison of WGDSF and traditional classic algorithms in the same experimental setting shows that the WGDSF has the best performance in most cases and has a good performance of the system.We design a database optimization scheme in the process of creating database tables and querying database. We make use of the column family independence storage’s characteristics in table creating process and optimize the table querying process for business research in HBase. We compare the radar data system performance with traditional Mysql database and the optimized HBase database and find out the HBase cluster is better.