| With the development of e-Science, scientific workflow has been widely used by scientists to perform complicated experiments and get important scientific discoveries. Due to the nature of science, scientific workflow often involves complex workflow design and distributed computation resources, so abnormal events are likely to happen and interrupt the normal execution of workflows. Thus, workflow monitoring and exception handling play a significant role within the context of scientific workflow. Machine learning pipelines are data pipelines which implement the tasks required during the machine learning application development. Scientific workflow could bring unique advantages when building machine learning pipelines.;In this dissertation, to tackle the challenges of workflow monitoring and exception handling, we propose a scientific workflow monitoring model and several workflow monitoring algorithms to realize efficient and effective workflow monitoring. We also propose architecture for workflow monitoring in DATAVIEW. Then we propose a user-defined exception handling framework for DATAVIEW, including a scientific workflow exception handling language, two exception handling algorithms as well as the exception handling architecture in DATAVIEW. At last, we showcase a case study using DATAVIEW to analyze NYC Citi Bike data by building machine learning pipelines using scientific workflows. |
