The Design And Implementation Of Workflow Management System Based On Transwarp Big Data Platform

When doing data analysis and processing,one job often fails to achieve our business goals.During the execution process,multiple job tasks are usually involved.These jobs are connected according to logical execution relationships,and there are interdependencies.If the current system increases the amount of data every day due to the transaction,the business department requires a statistical process every day,so the above process needs to be performed in a daily cycle.In this case,systematic scientific operation management is indispensable.In order to simplify the job scheduling,reduce the workload,and achieve automated workflow management,Transwarp released the workflow scheduling platform,The Transwarp Workflow.The Transwarp Workflow system described in this thesis is a graphical tool for implementing workflow scheduling DAG design and management.It supports task types such as Shell,SQL,and custom Java tasks.After the user sets the logical execution order of the task execution,it can automatically run the offline batch task on the workflow.The main programming language in the project is Java.The whole micro-service-based approach is quickly built through the Spring Boot framework.Spring's face-oriented programming and dependency injection enable low coupling between modules and high scalability.The backend service is designed by using the SpringMVC framework,and it provide a RESTful interface to communicate with the front through the RESTful interface,and use the MyBatis persistence layer framework to manage the mapping between Java beans and MySQL table records.In the development process,I am mainly responsible for the design and writing of some modules of the background program.This thesis mainly describes the related design and implementation of Transwarp Workflow system.The system is functionally divided into rights management module,task configuration module,task generate module,task scheduling module,notification module,log query module,statistical analysis module,and agent module.This thesis first describes the industry background of the workflow scheduling tool and the project backg round of the Workflow system;then introduces the data storage technology and system development technology used by the Workflow system;then describes the requirements and task-related module analysis and design of the Workflow system.The database design is introduced;then some system tests of the system are described.Finally,this thesis describes the implementation and results of the task schedule system module.