| With the development of the big data,researchers are paying more attention on the technology of processing large data set.In order to meet the growing demand for data storage and processing,more and more distributed applications are developed.MapReduce,proposed by Google,and its open source implementation Hadoop are capable of batch processing in distributed environment,and have excellent scalability and fault tolerance.By simply expanding MapReduce,Spark can effectively meet the performance requirements of most of the tasks because of the high-efficiency of data sharing primitives,which called the Resilient Distributed Dataset(RDD).In some application fields,the Spark is close to or exceeds the proprietary system performance,while also providing a more powerful fault tolerance capability and allows the interaction between assignments.With the construction of"Array for real-time geostrophic oceanography"(Argo),it has accumulated a large number of Argo environment observation data.However,the amount of Argo observation data is huge and it is discrete,so the efficiency of data retrieval in traditional way is too low.And some actual geographic applications require uniform and continuous distributed data to facilitate data visualization and further analysis,so appropriate interpolation algorithm is needed.However,because of the huge data and the complex interpolation algorithm,the interpolation calculation is very time consuming.Therefore,it is necessary to use distributed programming framework such as MapReduce and Spark for distributed processing to improve the interpolation efficiency.Based on the National Marine Cloud Computing project,this thesis focuses on the deep research of query processing and interpolation of the Argo data.The main works can be summarized as follows:(1)When the amount of the Argo data is huge,the traditional retrieval method will encounter the low efficiency problem,so a double-layer grid index suitable for the cloud environment is proposed.First,the original Argo data is stored on the HDFS after preprocessing.According to the features and actual demands of Argo data,a double-layer grid index structure based on hash partitioning and grid partitioning is proposed.The index is flat and suitable for distributed processing,which can effectively improve the query efficiency of data.(2)To meet the query demands of Argo data under the cloud environment,this thesis presents a variety of parallel query algorithm based on Spark according to the established index structure,including ID query algorithm,spatial range query algorithm and spatio-temporal range query algorithm,making full use of the filter function of double-layer grid index and the distributed parallel processing ability of Spark,achieving a good query efficiency.(3)Because of the complexity of the interpolation algorithm,the process of interpolation can be very slow when deal with large amount of data,so the current interpolation algorithms are transplanted to the Spark framework to realize distributed processing.With the PIRPS algorithm and hybrid reference point selection strategy,using Spark framework to realize the distributed processing of interpolation calculation can efficiently improve the efficiency of interpolation.Two common spatial interpolation algorithms based on Spark are implemented in this thesis.(4)Finally,extensive experiments have been done to evaluate the efficiency of query and interpolation processing of Argo data based on Spark.The experimental results showed that the index structure and related algorithms based on Spark framework can efficiently improve the efficiency of query processing of Argo data,and the efficiency of the improved interpolation algorithm is better than the original algorithm. |
