Research On Scheduling Method Of Data-intensive Workflow Based On Process Segmentation

With the rapid development of Internet of Things technology,data collected by various Internet of Things devices has been continuously generated and accumulated.These data have huge utilization value and generate a large number of business application requirements.Such applications have also become hot spots in many industries.In the application construction in these fields,many applications around these Io T data can often be expressed as a workflow composed of a set of big data processing and analysis tasks in the distributed computing environment of the Io T.This workflow is called as data-intensive workflow in this paper.Different from the traditional workflow,this data-intensive workflow in the Io T environment has the characteristics of scattered data sources,large data scale,and coordinated cloud-edge distributed execution.When executing such workflows in an Io T cloud-edge collaborative computing environment(hereinafter referred to as the cloud-edge environment),many factors such as business constraints and long-distance data transmission need to be considered,which brings many challenges to the execution engine of the workflow system in terms of data flow control management and data transmission scheduling.To this end,Aiming at the execution constraints and data transmission problems of this data-intensive workflow,this paper proposes a dataintensive workflow scheduling method based on process segmentation in the cloudedge environment,and designs and implements a data-intensive workflow execution scheduling system.The specific work includes the following three aspects:(1)Aiming at the execution problem of Io T data application,a data-intensive workflow scheduling framework in cloud-edge environment is proposed.For the application of Io T data,this paper uses the current mainstream business process modeling rule BPMN for workflow modeling,so as to facilitate the normalization of data-intensive workflows.In addition,The node definition of the cloud-edge collaborative environment and the main idea of the workflow scheduling framework in this environment are proposed,that is,to satisfy constraints and optimize data transmission through segmentation and scheduling.(2)By analyzing the workflow scheduling problem in the cloud-edge environment of the Io T,taking the user's constraints as the premise and the data transmission volume in the workflow execution as the optimization goal,a data-intensive workflow scheduling method based on process segmentation(DSPS)is proposed.First,determine the split position of the workflow under user constraints,and divide the original workflow into multiple sub-workflows according to the split position to reduce the data transmission between them and achieve the purpose of optimizing the transmission delay;then,The particle swarm optimization algorithm is used for improvement,so that it can meet the two optimization goals of workflow execution time and computing node load while satisfying user constraints.The results of the simulation experiments show that the research method in this paper has a good effect on the optimization of the overall execution time and the load of the computing nodes compared with the existing work.(3)On the basis of the above scheduling method,a data-intensive workflow execution scheduling system is designed and implemented based on the open source Flowable workflow software.Firstly,The architecture design of the system and the core process design of the system are given;then,the technical realization of the main functional modules of data-intensive workflow segmentation and scheduling is explained,and the visualization of the scheduling execution results is given;Finally,the usability of the system is verified through the scenario cases in the field of intelligent transportation,and the application effect of the system is shown.