Research On Hadoop-based MeteCloud Resource Storage And Data Processing

Currently, the meteorological bureau of each prefecture-level city has its own business systems and data storage systems. So it’s difficult to make the centralized management for the resources and realize resources sharing. A platform for sharing meteorological internal resources is in desperate needed. And now cloud computing can solve this problem with its rapid development after put forward.On the basis of analyzing the theoretical model of cloud platform this paper studied the meteorological daily data sets from the year1951to2012which was provided by International exchange station of Chinese surface data and emphatically has done the following works:(1) This paper first analyzes the architecture and the process of data reading and writing of the HDFS which is the distributed file system of the open source cloud platform Hadoop, and then studies the data processing of the computational model MapReduce, at last, does some researches with the distributed database HBase and describes the process of creating tables, and then studies the architecture and the process of data storing and query for the Hive warehouse.(2) The meteorology cloud platform MeteCloud is proposed with the building process and deployment of the clusters. The architecture of the MeteCloud includes hardware laye、platform laye、application layer and users layer. The Facebook AvatarNode mechanism as well with its working mechanism and running cycle is also introduced to solve the single point failure problem.(3) The transfer of static meteorological daily data on the platform of MeteCloud is also studied. In this paper, we analyzes the process of using Hive to transfer the meteorological daily data and we call it HiveDaily, and then analyzes the process of the using HBase to transfer the data and we call it HBaseDaily. Meanwhile, we also propose an optimizing process of using HBase to transfer the data base on MapReduce and we call it MRHBaseDaily (MapReduce-based HBaseDaily) to make the transfer process be more efficiently.(4) We also do some researches on the processing of meteorological daily data based on the MapReduce. The process of handling meteorological daily data based on MapReduce which called MRSMT (MapReduce-based SMT) is also proposed based on analyzing the process of SMT (Statistics of Maximum of Temperature) in the traditional local file system.On the basis of building the MeteCloud experimental platform, we do some experiments on the transferring of meteorological daily data and the acquired results prove that it can improve the efficiency for the meterological data storage and processing by using MeteCloud platform, and the optimizing storage process of HBase and MRSMT can also enhance the efficiency of transferring and processing data.