Design And Implementation Of Hadoop-based Workflow Management System

With the rapidly development of cloud computing and the rising demand of massive data computing, workflow management system has been drawing more and more attention as it plays an important connection part between customer and cloud computing resources. These systems, however, works as an independent third-party to the distribute systems like Hadoop. Although this construction makes it easier to manage workflow applications, it leaves no space for the optimization of workflow executions, besides, it gives us two few trigger mechanisms and many other urgent issues to solve.In this paper, we presented a new workflow system architecture that consists of a workflow engine and an built-in pluggable module for Hadoop, though which the system can detect workflow jobs, parse workflow information and manage workflow jobs’ scheduling.We also presented a workflow definition language called iPDL that realizes the workflow information transmission between workflow engine and the cluster, then, we showed the detailed grammar of the language that describe the workflow structure and configuration.To make full use of the advantage of the new system architecture, we also introduced some new workflow scheduling algorithm and Hadoop jobs batch scheduling algorithm. Finally, by using Hadoop simulators and actual deployment in clusters, we evaluated the speedup the these algorithms, the effectiveness of system functions, and the performance improvement of the system.