Research On Backpressure Mechanism In Storm

Under the background of big data,real-time computing and streaming data processing provide effective means for real-time applications such as real-time recommendation,real-time heat statistics,real-time stock trading and so on.Storm real time computing framework is an open source and distributed computing system.Because of the characteristics of high scalability,high fault tolerance and high efficiency,it has been received the attention of many scientific research institutions.And the framework has been widely applied by many large Internet companies in domestic and abroad.This thesis studies the key source code of Storm related technology,and analyzes the architecture of Storm framework,the components of Storm cluster and its running mechanism.On this basis,this thesis completed the following works:First of all,according to the problem that Storm default back pressure mechanism would produce data flow shocks,reducing system throughput and performance,this thesis puts forward to a back pressure mechanism that could adjust the data load of each link in Topology in a flexible way.The mechanism adopts variable queue,and according to the current Tuple load,adjusts the queue size dynamically in order to adapt to the dynamic changes of data load and improves the system throughput.The experimental results show that the mechanism can avoid the shock of data flow in the process of back pressure,and improves the performance and stability of the system.Secondly,this thesis puts forward the strategy of pre-backpressure based on the data flow prediction,which is used to deal with the waste of resources in the back pressure mechanism lagging.According to the self-similarity of the current data stream,the proposed method is used to predict the data flow.Then according to the calculation ability of the current computing nodes,the pre-backpressure strategy is adopted.The experimental results show that the proposed method has good accuracy,it can reduce the average processing delay of Storm cluster,improve the system throughput,and improve the system performance.The research results show that the research and improvement of Storm back pressure mechanism,could improve the average delay problem of the operation process of Storm tasks in a certain extent,and improve the throughput of the Storm cluster effectively,so as to improve the overall performance of Storm real-time computing framework.