Design And Implementation Of Fine-grained Asynchronous State Migration Strategy Based On FLINK

Distributed stream processing engines can be divided into stateful and stateless based on the characteristics of their operators;in the stateful distributed stream processing engine based on data parallelism,If the data is greatly skewed,the computing load of each processing node will appear unbalanced,which will affect the throughput and processing delay of the system.By changing the routing strategy,the data can be more evenly distributed to different processing nodes,and the processing load of each node can be balanced.In this process,since the processing node is stateful,after changing the routing strategy,the state in the node must be migrated accordingly to ensure the consistency of the system state before and after the adjustment.The existing state transition methods of stateful stream processing platforms mainly include: stop and restart,input replication,and state externalization.Each of these migration methods exists insufficient.First,the method of stopping restarting will cause delay spikes to the runtime system;second,the method of input copying will take up a lot of extra system resources;third,State externalization affects the rate of data processing.In response to the above-mentioned problems,this paper proposes a fine-grained asynchronous state migration mechanism applied in load adjustment scenarios to optimize the migration process in three aspects.First,the migration process is controlled asynchronously by an external controller,without stopping the system operation,avoiding delay spikes;Second,a routing table update algorithm is proposed,which optimizes the selection of migration target instances and further reduces the migration process overhead;Third,a fine-grained state splitting strategy is proposed.Through this algorithm,the number of migrations N under the minimized migration overhead can be obtained,and the complete state that needs to be migrated is divided into N times Migration greatly reduces the delay of data processing.The main goal of this paper is to in a stateful distributed stream processing system,without changing the parallelism of processing nodes at runtime,to reduce the overhead of system load adjustment and ensure the consistency of the system state before and after adjustment through a migration mechanism with less overhead.This paper analyzes and studies the entire load adjustment process of load tilt detection,routing strategy change,and asynchronous state transition.The cost of the migration process is modeled from the perspective of delay,and the key factors affecting the migration cost are obtained.According to the analysis of these factors,a routing table update algorithm and a finegrained split strategy are proposed to minimize the migration cost,which minimize the total amount of state transitions and the amount of single state transitions.