Test Strategy And System Optimization Of Remote Sensing Big Data Processing System

With the arrival of the era of global observations, satellite remote sensing information plays amajor role in promoting the development of human society in various fields. Because of thecontinuous improvement of the sensor resolution, the growth of the daily production of remotesensing images are rapidly. As a special kind of data Remote sensing images occupy anincreasingly important position with its quick access to the characteristics of high-resolutionimages. Above all of that, Mass remote sensing data processing has become more and moreoverwhelming demand. Against application needs of large-scale GF remote sensing data,multi-industry and multi-regional, China proposed16major projects in “development plan(2006-2020) of the national medium and long term science and technology” of State Council.High-resolution Earth Observing System is one of the major projects. And GF information product“production line” system is an important part of High-resolution Earth Observation System. Theauthor involve in the project and responsible for the original remote sensing data processingsystem. This article is the extraction of the work such as test, analysis and optimizes thearchitecture of the project. This paper contributes to accelerating the processing speed of the massremote sensing image.First of all, the article set testing strategies according to the characteristics of the originalremote sensing system and carry out the testing and analysis of the system effectively. ChapterTwo analyzes the architecture, technical process, operating environment, components and functionof the original remote sensing data processing system. On the basis of the analysis, Chapter Three design and plan the testing strategies of the original system, according to the characteristics offunctionality and performance about the cluster remote sensing processing system. Then testenvironment, test methods and test cases are selected strictly. In the addition, this article tests theperformance of the system such as task scheduling mechanism as well as GPU acceleration ratio.The test cases of the test strategies are clear, performance indicators are detailed, the target areclear and efficiency of the test is improved. Through the test we found the inadequacies and thetechnology need to be improved in the existing system, provided reference for subsequentarchitecture optimization, and guaranteed the system performance.Secondly, this article design a new kind of architecture-hardware and software integrationprocessing system of mass remote sensing image. Remote sensing data processing system havegood production capacity through testing the original system, but for remote sensing image-basedsystem is still relatively immature, there are still have problem about the response time andcomputing speed. Based on the test and analysis of the original system the article proposes anoptimization plan of improving hardware and software. This new architecture design the system’sdeployment scheme such as infiniband, fiber optic switches and the little giant. And the planre-designs the system process and hardware structure. Simultaneously, the effectively control ofthe data flow, optimization of GPU processing process, rapid storage and other new technologiesare integrated into the optimized architecture. The architecture is enable to meet the needs ofhigh-capacity, high I/O bandwidth and highly scalable of remote sensing processing system.Finally, this article takes the simulation test and simulation tests of the new architecture fromthe perspective of function and performance respectively. Additionally, it compares and analysesof the target of the original system and the target of the optimized system. The targets include the the comparison of order-processing time, comparison of concurrent response time of multi-task,the comparison of subtasks time executed by parallel computing nodes. The article also evaluatesthe ease of use, stability, reliability and adaptive capacity of the new architecture. Through testingand evaluation, we can see that the new architecture use the cluster resources efficiently andreasonably. The time that subtasks produce product executed by every parallel computing nodestabilize between4and6seconds. And the speed processed by GPU is40times than the speedprocessed by CPU. Also concurrent multi-task response time is78.12%higher in average thanbefore and the data transmission rate is78.9%higher in average than before. The architecturewhich is optimized has the advantage of large storage capacity, high transmission bandwidth, fastresponse, update frequency and well-equipped functions. Therefore, the architecture not onlymakes a connection with development trends of information technology closely but also ensuresthe correct design and selection of the architecture and provides a guarantee for the expansion offuture maintenance and testing in various stages.