Optimization Of Large Scale Of Files Transfer In Meteorological Grid

The meteorological data grid is one of the key projects of National Meteorological Information Center. The performance of observation data’s transmission, sharing and comprehensive service is still relatively low. Thus the data’s timeliness, category and completeness are not enough to satisfy the demand of elaborative forecast, numerical forecast, impending prediction and others. This paper aims at optimizing the data transmission services to meet the demand of real-time data sharing under the background of national meteorological data grid.The optimization method of data transmission presented in this paper focuses on the issues of network transmission protocols and file compression technology, supplemented with a distributed cache technology. And, at the end of this paper, it gives a feasible optimization scheme. The main research contents are as follows:(1) It gives a brief summary of researches on the optimization methods of data transmission at home and abroad, and we do our own research in the aspects of network transmission protocol, file compression technology and distributed cache technology. Further, the advantages and disadvantages of the existing optimization methods are summarized based on lots of literature. What needs to note is that it takes much effort to analyze the existing GridFTP protocol optimization algorithm, LZW and BZIP2compression algorithm and distributed caching technology.(2) It proposes the adaptive data compression algorithm. First, it gives the compression principle, characteristics, advantages and usage after analyze LZO compression algorithm in detail. Second, it summarizes the experimental analysis and formula of meteorological files, thus proves the basis that the weather files are belong to a large number of small files. Third, it analyzes acquisition method of real-time network status, sets the processing mode of the status of network fluctuation, and quantizes real-time network fluctuation. Finally, it gives the design of algorithm with self-adaption compression level based on real-time network status, and the algorithm flow chart, experiment results and analysis are also included.(3) Design the self-adaption transmission parameter tuning algorithm. First, this chapter analyzes the GridFTP protocol, and describes its characteristics, application area, and service components. Second, it finish the two sets of experiments, verify that the TCP buffer size and TCP concurrent connection number have its optimal value in different network environment, and illustrates the importance of GridFTP protocol transmission parameters. Third, according to the real-time network status, it gives the self-adaption adjustment algorithm of TCP buffer size and the concurrent connection number algorithm of TCP based on the above analysis respectively, and verifies it by experiments. Finally, carry out complete experiment of the integration of self-adaption algorithm and analyzes the experiment result by combining the content of several self-adaption algorithms.(4) Implement distributed cache technology based on improved cache replacement algorithm. First, it analyzes the characteristics and implementation of EHCache and Terracotta. Second, it gives the design of improved cache replacement algorithm OUR based on the specific application context referred in this paper. Third, it completes coarse-grained distributed cache design based on opensource project of EHCache and Terracotta. Finally, verify the availability of improved distributed cache framework after adjustment by several experiments.