The Design And Implementation Of Airflow Scheduler Cluster Based On Raft

With the increasing amount of data and complexity of business requirements,developers begin to execute their operation flow with the assistance of task scheduling system.As a user-friendly distributed task scheduling framework with abundant functions provided,Airflow has been approved and selected by many enterprises and developers.However,as a distributed service,Airflow has few defects that can not be ignored.For example,scheduler node in Airflow is only able to be deployed in single point mode,and the monitoring and management for users is complicated when cluster is deployed in large scale.Aiming at these defects,the Airflow Scheduler Cluster System is designed and implemented in this thesis after investigation of current high-availability solutions.This system supports the clustering deployment of scheduler nodes,and grants faulttolerance ability for scheduler nodes based on Raft consensus algorithm,thereby greatly improving the availability of cluster.To ensure consistency for Airflow cluster,the TCP requests generated from Airflow scheduler process are monitored and restricted.Furthermore,this system provides a cluster administration website and realizes configuration file version control to simplify the operations for users during the maintaining and managing process,which will improve the usability of Airflow cluster.In this thesis,defects of current Airflow are analyzed first and then requirements of the Airflow scheduler cluster system are clarified.According to the requirements,the system is designed into four main modules,namely Raft module,TCP proxy module,configuration file version control module and cluster administration website module.Next,the whole system is implemented in Python.IO multiplexing is introduced in the implement of a high performance TCP proxy.The cluster administration website and interfaces of configuration version control are implemented based on Flask framework.Finally,a series of experiments were conducted to test the availability and functionality of the proposed system.Experiment results show that Airflow scheduler node can be deployed in cluster with the support of the Airflow Scheduler Cluster System,and can greatly improve the fault-tolerance ability.Meanwhile,its performance in cluster management has also met our expectation.