The Research Of Task Scheduling And Load Balancing Mechanism Of Storm Real-Time Computing

Under the background of big data, data contains a wealth of meaning and value, and hysteresis data analysis will affect the value of the data. In the real time computation and data stream processing is becoming more and more important today, Storm came into the framework of real-time computing. Storm is an open source real-time distributed computing system, it designs an abstraction of stream and computing. Storm real-time computing framework with its high scalability, high fault-tolerant, high efficiency and other good characteristics, has been favored by many scientific research institutions and has been widely used in data assets rich enterprise at home and abroad.By studying the Storm related source code, this thesis focuses on the analysis of the system architecture of Storm system, the implementation mechanism of Storm scheduling algorithm and the load balancing strategy of Storm cluster. On the basis of this, the method of improving Storm scheduling algorithms and optimizing cluster load balancing had beeb researched. Specifically, the thesis mainly completed the following work:1. In view of the problem that Storm default scheduling algorithm does not take communication pattern of the component into consideration, which causes the problem of long communication delay, the Topology hot-edge based scheduling algorithm has been proposed, scheduling hot edge associated executor to the same node. Experimental results show that the scheduling algorithm can effectively reduce the inter-node communication, shorten tuple processing delay and improve system throughput. Compared to the default scheduling algorithm, the improved scheduling algorithm promotes 20%-25% of the overall performance.2. For the problem of imbalance useage of inter node cluster slot resources, the slot utilization rate low priority based dynamic load balancing strategy has been proposed, which gives priority to the nodes with low usage rate of slot and dynamic transfer load among cluster nodes. The experimental results show that the load balancing strategy can effectively balance the slot usage between nodes, and promote the load balancing of Storm cluster system.The research work shows that the research and improvement of Storm scheduling algorithm and load balancing strategy can further improve the communication delay of Storm processing tasks and make the slot utilization of Storm cluster node more balanced, so as to enhance the comprehensive performance of the Storm real-time computation framework.