Space-Ground Integrated Optimization Approach For Earth Observation Big Data Cloud Service

Confronted with rapidly increasing satellite imaging resolution and increasing satel-lites in orbit,the speed and total volume of data acquired by space-based Earth observation satellite systems are increasing explosively.According to reports,currently the volume of Earth observation data acquired in China is up to hundreds of TB per day,and the growth momentum is strong,which means that the era of Earth observation big data has arrived.At present,users have higher requirements for satellite data services.In terms of time resolution,the users urgently need the response during the whole process of acqui-sition,transmission,processing and distribution for satellite observation data to become nearly real-time;for spatial resolution,users require that the resolution of satellite im-ages is up to sub-meter or even centimeter,and the observation range covers the whole world.Especially when facing the scenarios including time-sensitive targets,maritime mobile targets,and earthquake relief operations,users have higher demands on the time and spatial resolutions for satellite Earth observation data services.System and management isolation of current Earth observation satellites and com-munication satellite systems leads to slow response to Earth observation data services,low resource utilization,and it is difficult to meet the dynamic high concurrency of high time-space resolution service requirements worldwide.To improve satellite coverage,re-duce response time,and improve system synergy,it is necessary to form an integrated platform for satellite networks and terrestrial cloud computing.Then,driven by tasks,the resources dealing with satellite data acquisition,transmission,computing,storage,and data processing are collaboratively organized and optimized.Using Earth observation data to service users directly as the core,open up the fast service thoroughfare initiated by the user,via data acquisition,transmission,processing,intelligence extraction and distri-bution sharing,and user application.This paper focuses on the following five key issues:task scheduling for the acquisition of Earth observation data,optimizing inter-satellite data transmission in the stage of data transmission,real-time scheduling for big data pro-cessing,security management and uncertain optimization.By addressing the above key issues,this paper mainly makes the following five contributions:(1)An adaptive multi-objective evolutionary algorithm is designed for satellite schedul-ingToward the multi-objective scheduling problem of satellite Earth observation tasks,this work defines a new indicator to measure the contribution of each subspace to popu-lation convergence in multi-objective evolutionary algorithm,and proposes an adaptive strategy to adaptively allocate computing resources to different subspaces according to their contributions.On the basis of the above two strategies,this work designs an adap-tive multi-objective optimization algorithm based on objective space partitioning,which accelerates the convergence speed of the population while maintaining its diversity,so as to quickly search a set of effective solutions to balance multiple conflicting objectives for scheduling satellite Earth observation tasks.(2)A self-organizing optimization approach is proposed for offloading Earth obser-vation big dataAiming at the randomness and suddenness of Earth observation data,as well as the high dynamic inter-satellite topology in resource restricted satellite networks,this work proposes a self-organizing optimization approach for offloading Earth observation big data in satellite networks,such downloading these data to the ground station in near-real-time.Specifically,the approach defines satellite gradients to fully reflect the relationship between available resources and constraints,and designs a new strategy to update the neighborhood of each satellite to handle the high dynamics of the satellite topology.Based on the satellite gradient and neighborhood,this paper proposes an optimization strategy based on bidirectional selection,which supports each satellite to make decisions on data offloading during interaction with neighboring satellites.(3)A real-time scheduling method is developed for Earth observation big data pro-cessing in cloud computingEarth observation big data continuously streams to ground stations and are dynam-ically submitted to cloud computing for real-time or near real-time processing.The ran-domness and suddenness of the Earth observation data stream seriously challenge the rapid response capability of cloud computing platforms.To solve the above problem,this work derives two lemmas to minimize the completion time of a set of tasks and the start time of each workflow task.Then,this work defines the latest completion time for the workflow task and demonstrates that the latest completion time helps reduce the cost of Earth observation big data processing without delaying their completion time.Further,this work proposes a task duplication-based scheduling algorithm to deploy big data streams to the cloud computing platform for real-time processing,minimizing the response time of cloud computing and reducing the cost of big data stream processing.(4)A security-aware scheduling approach is proposed for Earth observation big data processing in cloud computingData encryption is a promising way to ensure the security for security-sensitive Earth observation big data.However,the time overhead of data encryption inevitably delays the completion time and increases cost of big data processing.To solve the above problems,this work first analyzes how to duplicate tasks to alleviate the delays of data transmission and encryption on task start time,and then proposes a scheduling method for security-sensitive workflow,which contains two important stages:task scheduling with selec-tively duplicating predecessor tasks to idle time slots on resources;and intermediate data encrypting by effectively exploiting tasks'laxity time.(5)An uncertainty-aware scheduling method is proposed for Earth observation big data processing in cloud computingFocusing on the serious impact of uncertain factors on the performance of cloud computing platform processing Earth observation big data,this work designs a novel scheduling architecture to control the number of waiting tasks on each service instance to prevent the propagation of uncertainty.Based on this architecture,this work proposes an uncertainty-aware online algorithm to schedule big data processing workflows with deadlines.The algorithm skillfully integrates both the proactive and reactive strategies.During the execution of the generated baseline schedules,the reactive strategy will be dynamically called to produce new proactive baseline schedules for dealing with uncer-tainties.