Research And Implementation Of Storm Energy Saving Scheduling Strategy Based On Energy Consumption Perception

With the rapid development of information technology,the data generated by the Internet has exploded in recent years,and big data processing frameworks such as Storm,Spark and S4 have also emerged in this context.The proliferation of data has also brought about an increase in the energy consumption of processing data,And Storm is the mainstream real-time processing framework for big data,the energy-saving scheduling of the Storm platform is of great significance for big data energy saving.This thesis deeply studies and analyzes the composition and working principle of Storm,and elaborates on the related research of Storm's scheduling algorithm and big data energy saving.Storm's native scheduling algorithm does not consider energy consumption as a scheduling consideration.In the actual task processing process,due to the difference in resource requirements,the task may bring too much energy waste.Aiming at the above problems,the corresponding optimal scheduling algorithm is proposed and verified by experiments.The main work of the thesis is as follows:1.A Storm energy-saving scheduling algorithm based on energy consumption perception is proposed.The energy consumption monitoring module and the persistence module are added to the Storm architecture,and the energy consumption of the node processing tasks is sorted,and the energy consumption of the cluster processing tasks is reduced by integrating the tasks into the low energy consumption nodes.Through experimental comparison,under the condition of processing the same amount of data,the energy-aware-based Storm energy-saving scheduling algorithm reduces the total energy consumption by about 35%compared with the default scheduling algorithm.2.An improved energy-aware Storm energy-saving scheduling strategy is proposed.By sorting the Slot utilization of low energy consumption nodes in the cluster and assigning priority to the low Slot utilization nodes for task processing,the cluster load is more balanced and the overall performance of Storm is improved.The improved energy-saving scheduling strategy can satisfy the SLA(Service Level Agreement)and reduce the total energy consumption of cluster processing tasks.Compared with the default scheduling algorithm,the total energy consumption of cluster processing tasks is reduced by about 25%.At the same time,the load state of cluster and the completion time of processing tasks are better than the energy-saving scheduling algorithm based on energy consumption perception.The research work in this thesis shows that the improved scheduling algorithm can effectively reduce the total energy consumption of the Storm cluster processing tasks,and can reduce the total energy consumption of the processing tasks while satisfying the SLA conditions,which is of great significance for big data energy saving.