Research And Implementation Of Key Technologies For Big Data Workflow Systems

Business Process Management(BPM)is a systematic approach centered on the disciplined construction of end-to-end business process excellence with the goal of continuously improving organizational business performance.BPM is used in various fields,including Science and technology service.In the current social environment where technology industry is emerging and service demands and types are diversified,business workflow can scientifically express and describe various business processes in the field of technology services,enhance the standardization and flexibility of business processes,and achieve the purpose of improving informationization,business execution efficiency,and efficient collaboration with multiple enterprises or service providers in the field of technology services.At the same time,a huge amount of data will be generated in the field of technology services,and the reasonable application of these data can better provide personalized services for enterprises in the field and further promote the development of the field.However,if the big data businesses processes can be managed using the traditional BPM approach,although the execution efficiency and automation of the big data processes can be improved,there are certain limitations at present.On the one hand,The existing workflow modeling standards do not support building big data business processes.The big data business process can only be divided into multiple human tasks before execution,which is not friendly to non-technical personnel and can lead to non-standard and poor reusability of the business process,resulting in reduced efficiency in completing the business process.On the other hand,the execution of workflow processes supporting big data technology will increase the resource consumption and load of the workflow engine,leading to a decrease in process execution efficiency.To address the above problems,the specific research work in this paper is as follows.(1)A business process modeling method supporting big data technology is proposed.According to the modeling requirements of Big Data business processes,the metamodel in BPMN2.0 is extended to express Big Data processes by defining five kinds of Task elements under this class,and the XML schema definition of each Task is extended.The XML schema definition of each Task is extended so that the Big Data business process can be efficiently built into a Big Data workflow process model by business process management.The feasibility of the modeling method is verified by taking the big data business process in the field of technology service as an example.(2)A big data-enabled workflow federal scheduling method is proposed.In collaboration with the business-oriented workflow execution engine Flowable and big data processing systems(Hadoop,Spark),we constructs a federated workflow execution scheduling management method,in which the Flowable business-oriented workflow engine is responsible for scheduling and executing BPMN2.0 regular tasks and controlling the execution progress of process instances;the big data tasks are assigned to the distributed big data processing system Big data tasks are assigned to distributed big data processing system,and the results of big data processing are fed back through message middleware,so as to achieve the overall execution efficiency of the workflow system by having multiple types of technology platforms doing their own jobs.At the same time,in order to improve the performance of workflow systems and reduce workflow execution time,based on the complexity of big data business processes,as well as the number and sparsity of big data tasks in the business process diagram.This article proposes two big data task scheduling strategies,which allocate big data tasks to appropriate data processing server nodes according to the scheduling strategy based on the workflow process model before executing big data tasks.Meanwhile,through experimental evaluation and analysis,it is proved that the scheduling method proposed in this paper can effectively improve the efficiency of workflow execution and the correctness of algorithm selection under different scheduling strategies is demonstrated.(3)The distributed workflow system is designed and implemented.The work of this article is implemented in the Saa S platform for technology service collaboration under the project "Research on Key Technologies of Distributed Technology Service Application",and the business process modeling method supporting big data technology and the federal scheduling method of workflow enabled by big data proposed in this paper are applied.Expand business process modeling based on Flowable business workflow system and BPMN.JS;Build Hadoop and Spark big data platforms to support the execution of big data tasks in workflows,apply Kafka message middleware for communication between big data platforms and business workflow systems,and use system application schedulers to improve workflow execution efficiency.This paper proposes a business process modeling specification that supports Big Data technology to address the problem that the current workflow modeling specification does not support Big Data business processes;and proposes a Big Data-enabled workflow federal scheduling method to address the problem that executing Big Data workflow processes increases the resource consumption of the workflow engine and reduces the process execution efficiency.The above two key methods are applied to the distributed workflow system built in this paper,which can describe and execute the big data business processes in the field of technology services in a standardized and efficient way.