Parallel Computing Of Spark-based Geospatial Analysis Algorithms

With the rapid development of earth observation technology,the spatial data obtained by people has increased exponentially.As a typical implementation of high-performance computing,parallel computation technique refers to using multiple resources to solve computing problems at the physical level at the same time,or decomposing the algorithm into several modules that can be executed at the same time from the logical level.Its purpose is to improve the computing speed and expand the scale of processing problems.In contrast,the parallelization of traditional GIS basic serial algorithm has become a hot spot of GIS researching.Most of the existing research on parallel spatial analysis is based on the algorithm level which is difficult to achieve,and often relies on specific platforms or specific algorithms.That is to say,their universality is poor,and there are also defects in the load balancing of the calculation on each node,which is easy to cause a waste of resources;the research on data parallel is relatively less.Because of its spatial and object attributes,geographic data can not be perfectly supported by native big data processing tools.How to combine big data processing tools with basic spatial analysis algorithms is also a key and difficult point in the research.In order to solve the problems of unbalanced load of computing task partition and insufficient quantification of computing load(mainly for computing time)in the process of parallelization of existing GIS basic spatial analysis algorithms,this paper mainly carried out the following work based on spark framework:First,a general parallel computing process for spatial analysis algorithm was proposed,The traditional spatial analysis algorithms are classified according to the degree of computational complexity,and the corresponding parallel computing flow scheme is designed for each class.Second,in order to divide the computing load evenly,a space coding method based on multi-level adaptive Hilbert curve is designed.Third,using the results of the third national land survey as the base of experimental data,combined with the parallel computing acceleration scheme proposed in this paper,a number of control experiments were carried out to verify the acceleration effect of the scheme and explore the best acceleration conditions;The data used in this paper is based on the results of the third national land survey of a county in Yunnan Province.After image affine transformation,the experimental data are obtained.The representative algorithms are selected in the strong/weak spatial-correlation algorithm for comparative experiments,and the experimental results show that the overall scheme achieves good acceleration effect.