Research On Load Balancing Technology Of Streaming Data Processing Platform JStorm

In the era of big data, the streaming characteristic of data is becoming more and more remarkable. A number of application scenarios are deployed on streaming data processing platform. However, with application scenarios becoming more complex and vigorous growth in data volume, imbalanced load among nodes in distributed computing platform has already become a bottleneck to restrict performance improvement of streaming application. Therefore, the value of research about load balancing issues on streaming data processing platform has become significant. This dissertation takes JStorm streaming data processing platform as the object of study and it devotes itself to resolve the imbalanced load of JStorm platform which is caused by inappropriate allocation of operation resource.This dissertation makes deep analysis on the strategy of resource allocation under JStorm platform and points out the clustering imbalanced load problem which may be caused by this distribution strategy of resource allocation under heterogeneous circumstance of available resource among pitch points. In addition, aiming at this problem, improved distribution strategy of resource allocation is put forward. Through considering the asymmetry of node workload and reasonable allocation of computational tasks, the operating performance of streaming application is enhanced. Main work accomplished by this dissertation include:1) This dissertation makes detailed introduction to such technologies as streaming data processing, JStorm streaming data processing platform and distributed platform resource allocation etc. It focused on analyzing the whole process of computational task in JStorm from its submission to decomposition into specific operation tasks. It also researches on the process of resource allocation by JStorm for operation and points out the clustering load imbalance resulted from lack of overall evaluation of computing resource in the process of resource allocation.2) This dissertation puts forward the load evaluation model for JStorm worknode. By researching present load evaluation model for node and combining the character of JStorm operation data processing, starting from computational resource utilization rate of JStorm worknode and Data processing capability of node itself, multi-weight load evaluation model for JStorm worknode is designed and plentiful concepts in the model are described in detail.3) The resource allocation strategy of JStorm operation has been improved. Aiming at the situation where the characteristics of unsymmetrical load of worknode is overlooked in resource allocation strategy of JStorm operation, in combination with the proposed worknode load evaluation model, the loading condition of worknode shall be taken into consideration in process of operation resource allocation and the computing resource must be distributed reasonably so that balanced cluster load is guaranteed and the streaming application is able to make full use of computational resource at each node.4) This dissertation designs and implements the improved strategy of resource allocation. It embeds the source code into dispatcher of JStorm, puts up a JStorm cluster to verifies the feasibility of improved strategy through experiment. The experimental result indicates that the improved resource allocation strategy which is proposed in this dissertation has solved the clustering load imbalance resulted from inappropriate allocation of computing resource in JStorm, at the same time, the data processing performance of JStorm is also improved, with data throughput being increased by 10%.