Load Management Functionality Design And Implementation In The Framework Of Distributed Stream Computing

Load management technology has an important theoretical and practical significancefor the heterogeneous distributed computing environment. At present the load balance technology in P2P architecture environment has long been a hot research, there have been many solutions to optimization P2P system load balance performance. P2P system is divided into two major categories of structured and unstructured, structured P2P systems based on distributed hash table (DHT) can solve the unstructured system resources orientation problem. However, in structured P2P systems, the requesting task and node through the consistency of the Hashi function is stochastic equilibrium is mapped to a logical ring, the load balancing problems caused bysome characteristics of the requesting task, which performance in real-time massive request task information, the vast majority of requests tasks there are hot issues, have become increasingly prominent. In order to effectively solve the request hotspot issues in the structured P2P system,existing programs to address how to reduce the load transfer in communication Overhead issues, which is the communication cost problem of hash ring value and processing node control relationship when changed. However, since the data processing request hotspot changeability,They did not consider the system to transfer the load fluctuation brought about by the data processing requests heat change, especially in the number of nodes with a certain scale environment,volatility will be more intense, the communication cost becomes large,which affect the system load balancing convergence rate and global load Balance. Again, these are not considered the overload control problems of structured P2P system in the case of all nodes full under the condition of overload,when the system atthe rationale for the requested amount exceed the maximum load of the system, will lead to the collapse of the whole system, affect the stability of the system and the disastrous consequences. In order to solve the problem of request hotspot, which brings loadtransferring,loadunbalancing and overloading, this paper proposes an improved dynamic load management algorithm.In order to solve the load balancing of structured P2P system on computing resources level, this paper proposes a mechanism,which adopts consistent hashing combination of virtual node strategy,to ensure that each node loadbalancingin the distribution system according to its comprehensive processing capacity.To deal with the request hotspot and its mutability, which cause the load fluctuation problem, the load transfer witha request redirection ideas is put forward to reduce the fluctuation of system and guarantee system load balance.And we use random discard strategy admission mechanismto handle overload problem. In order to verify our load management system can brings better load balance and the stability of the system to distributed stream computing system, based on S4system,which is an open source stream computing framework,this paperdisplay a distributed load management function design, implementation and verification.