| In the era of rapid data development,applying technology to process and analyze massive amounts of data can discover patterns and trends in the data,and combine them with other fields to help make decisions,thereby generating enormous value.With the increasing development of distributed data processing frameworks,operations such as data processing and analysis based on cluster frameworks have gradually become complex and diverse.Batch data processing is no longer sufficient to meet the current demand for real-time data applications,and data stream computing has gradually entered the field of vision and received widespread attention.This paper mainly studies the task scheduling algorithm of the Big data processing framework,selects Flink framework to build a distributed cluster,and designs a resource aware monitor based on the Flink cluster default task scheduling algorithm.Based on this,a polling task scheduling algorithm based on dynamic self-adjusting weight is proposed and implemented.This algorithm selects representative performance indicators to quantify cluster performance,and divides them into static performance indicators and dynamic performance indicators based on resource characteristics.After initializing the Flink cluster node weight configuration,periodically collect performance indicator data of cluster nodes,calculate node weights using the weight calculation model constructed in this article,and finally determine whether to update node weights and achieve node updates through calculation.This algorithm dynamically changes the weights of nodes in the Flink distributed cluster,and improves it based on the default task scheduling algorithm of the Flink cluster.This enables the cluster to allocate task requests based on the real-time performance of nodes,improves the utilization of cluster resources,and reduces system response latency.This article builds a heterogeneous distributed cluster testing environment and deploys a dynamic self adjusting weight polling task scheduling algorithm related model.Using the streaming application provided by Hibench as the workload,the experimental results are compared and analyzed.The experimental results show that the algorithm proposed in this article can effectively reduce data processing execution time,improve cluster throughput,and improve the overall performance of the cluster. |
