Research And Implementation Of Flink-based Elastic Scheduling Strategy In Constrained State

In pace with the develop at top speed of edge computing and real-time computing,there are more and more data real-time computing scenarios under the edge computing architecture,which requires the computing engine to have the ability to deal with the possible resource constrained state in edge computing and ensure the real-time performance of computing.As the most cutting-edge distributed flow computing engine,Flink is widely used in industry.However,according to the feedback from the industry,the real-time performance of Flink's tasks will be greatly affected in the edge computing scenario.It is found that the main reason is the default polling based responsive scheduling mechanism adopted by Flink.This mechanism leads to the time difference between scheduling and task requirements,which seriously affects the real-time performance of computing;At the same time,Flink also lacks an elastic scheduling mechanism.After the task is started,it is unable to flexibly adjust the resources according to the operation.Aiming at the problems existing in the existing Flink scheduling engine,this thesis proposes a Flink constrained elastic scheduling algorithm based on Weighted SVM:WSVM-Flink,and builds a Flink on Kubernetes elastic scheduling system based on this algorithm.In terms of algorithm,WSVM-Flink algorithm is based on the predictive elastic scheduling scheme,uses the weighted support vector machine(WSVM)algorithm to complete the resource prediction before scheduling,and formulates the fine-grained scheduling algorithm based on Flink layer according to the prediction results to change the resource allocation of Flink tasks in operation;In addition,the coarse-grained scheduling algorithm based on Kubernetes is formulated,and the resource elastic expansion module of Horizontal Pod Autoscaler(HPA)of Kubernetes at the bottom is combined to flexibly change the resource allocation of the cluster;At the same time,the solution when the resource limited state occurs is fully considered in the algorithm design,so that the WSVM-Flink can adapt to the resource limited state;In terms of system construction,Flink on Kubernetes Native cluster construction method is adopted,and WSVM-Flink is integrated to build an elastic scheduling system.The system includes monitoring module,prediction module and execution module,and has a complete data processing process.Finally,the performance of WSVM-Flink algorithm and elastic scheduling system is entirely checkouted.The tests result display that compared with other Flink elastic scheduling algorithms,WSVM-Flink performs better in terms of computational delay and resource consumption;At the same time,the flexible dispatching system has perfect functions,and all requires acquire the system designing demand.The research and implementation of the scheduling strategy immensely enhanced the realtime function of Flink,reduces resource consumption,and can be applied to the data real-time computing business in the limited state.