Research Of Distributed Remote Sensing Image Processing Based On Hadoop

With the development of Remote Sensing and aero photography techonology, there are more ways to acquire Remote Sensing data. And the size of remote sensing data is undergoing an explosive increase, due to the spatial, temporal and spectral resolution become much well than before. How to effectively and efficiently store, organize and process these data has become an advanced research hotspot in remote sensing field. For addressing these issues, distributed storage and parallel computing have been proposed.Recently, with the advantages of high reliability, scalability, efficiency and fault-tolerant, Hadoop have become the most popular and successful open source distributed system framework. The core content of Hadoop are HDFS and MapReduce. It is more and more popular to use Hadoop to solve the problem of the storage and computing mass data. Huge success has been made in SE (searching engine), E-commerce and Social Networking Services. GDAL, a set of open source geospatial data library, can extend support format by plug-ins. It also support to read and write most of the types of remote sensing image, the differences in image formats can be ignored. As a result, only one kind of code can process various image formats.In this paper, an image progressing program is developed, which combines GDAL with Hadoop, according to the mode of GDAL reading and writing images, analysis of HDFS’s storage characteristics and framework programming model of MapReduce. Specifically, the main achievement of this research is as follows:Firstly, analyzing the program model of MapReduce framework and how MapReduce preprocessing the text data, and designing a MapReduce library on the base of GDAL.Secondly, the efficiency of minimum distance classification for image under the condition of single PC, Hadoop cluster, and Hadoop cluster of different conditions are compared. Results show that the data amount of computation, cost of computation and machine cost have significant impact on the efficiency of remote sensing image process operation. Hence, the above three factors need to be taken into consideration before deciding whether to use Hadoop cluster and the size of cluster.Thirdly, Because Hadoop cluster could process single big file in higher efficiency than multi files of the same data amount, this paper designs a file format by merging multi images into one file in order to reduce the files count on HDFS and improve the utilization efficiency of metadata storage and searching time. Multi image processing with single MapReduce task is achieved and the utilization of every task is improved. An experiment of image projection transformation with MapReduce is conducted to test the practicability and high efficiency of the file format from aggregation large image.