Research On Container-based Scheduling Technology For Large-scale Atmospheric Remote Sensing Inversion

The rapid development of current satellite remote sensing technology has generated a huge amount of atmospheric remote sensing data.Most of the current methods for processing remote sensing data adopt the method of single-machine deployment of remote sensing algorithms,whose deployment configuration is cumbersome and difficult to meet the processing needs of large-scale data.The existing remote sensing algorithms are difficult to be rewritten internally due to the mobility of designers and the confidentiality of algorithms.To address the problem that traditional stand-alone remote sensing algorithms cannot be executed in a distributed environment,a container-based remote sensing algorithm image encapsulation and resource pre-allocation strategy is designed.To address the inefficiency of complex atmospheric remote sensing tasks in distributed cluster data processing,a remote sensing task scheduling algorithm with a priori values and a strategy of dynamically switching task scheduling algorithms are proposed.The main work includes.Firstly,we study remote sensing algorithm mirror encapsulation and remote sensing task scheduling strategy with a priori values.The traditional remote sensing algorithm runs directly on a single machine with low data processing efficiency and requires the installation of a dependency environment.In this thesis,the traditional remote sensing algorithm and the dependency environment are encapsulated into a mirror so that the algorithm is free from tedious configuration and has the feature of "once encapsulated,everywhere executed".In order to solve the problem of poor resource utilisation of remote sensing tasks using traditional scheduling algorithms,an atmospheric remote sensing task scheduling method with a priori values is proposed.The scheduling strategy with a priori values first constructs an a priori model based on the previous execution records of remote sensing tasks,and then calculates the priority of the task and generates task queues and allocates resources based on the a priori model values.It is found that the scheduling method with a priori values improves the accuracy of remote sensing task scheduling and the utilization of cluster resources.Secondly,the dynamic switching scheduling algorithm and remote sensing task distribution are investigated.In order to solve the complex remote sensing task scheduling requirements in a distributed environment,a dynamic switching scheduling algorithm strategy is proposed.As a global scheduling strategy,the dynamic switching scheduling algorithm strategy matches the key features of the task characteristics with those of the scheduling algorithm,and dynamically switches the scheduling algorithm to improve the flexibility of the system.The Spring Cloud technology is used to build a distributed cluster,and the node initialisation service is designed to enable the nodes to have the ability to execute commands.A large remote sensing task splitting strategy is proposed.By splitting a large task into multiple subtasks that can be executed in parallel,the method is experimentally verified to achieve the goal of distributed and fast execution of the original stand-alone tasks in the cluster system.Thirdly,the capability scheduling and dynamic scheduling strategies of tasks and data for remote sensing tasks are studied.A capability scheduling strategy for different remote sensing tasks is designed to address their different demands on cluster resources.The capability scheduling strategy divides system resources into multiple task resource queues,with the resources allocated within each queue differing from the scheduling algorithm,and allows the borrowing of resources from other queues when the resources within that queue are insufficient,further improving the flexibility of the system.A dynamic scheduling mechanism for tasks and data is proposed,which enables tasks and data to be scheduled again when a node fails or a task fails,and the method is found to be effective in enhancing the reliability of the system after experiments.Through the above theoretical research and experimental validation,a large-scale atmospheric remote sensing inversion scheduling platform was designed and built,and the overall design of the platform and the effect of the core functional modules were introduced and demonstrated.At the same time,the MOD021 KM data of the Beijing region was used for the calculation of AOD inversion products,and the correctness and rapidity of the platform in executing remote sensing tasks in a container manner were verified,which has good application and promotion value.