| Many distinctive features presented by streaming computing applications in a big data environment has been systematically analyzed in this dissertation.From the perspective of system architecture,a principled strategy for fault tolerance of big data streaming computing systems is given.Combined with the current typical Heron streaming computing platform,the fault tolerance mechanism of Heron streaming computing platform is mainly studied.The fault-tolerant mechanism of the Heron platform has been mainly improved from the two aspects of the Acker component thread allocation mechanism and the topology parameter update mechanism in this dissertation.First of all,Heron’s existing tuple tracking monitoring mechanism sets the number of Ack component threads to a fixed value.This strategy cannot adapt to the dynamic change of the number of tuples.When the number of tuples is too small,thread waste caused by too many threads in the Acker component.Too many tuples will cause high delays in communication messages due to excessive thread load on the Acker component.In order to make reasonable use of memory,a tuple tracking strategy from the perspective of energy saving has been proposed in this dissertation.The tuple is mapped to a hash table by a hash function,and the number of threads of the Acker component is dynamically adjusted according to the occupation of the hash table.The number of component threads is compatible with the size of the data flow in the current topology.Second,Heron’s existing topology parameter update mechanism requires programmers to select the cluster tuple timeout retransmission threshold based on experience.This strategy has certain risks and randomness.It may cause the problem of large recovery overhead.Based on the analysis of Heron topology attributes,a topology dynamic retransmission strategy is proposed.The topology dynamic retransmission strategy consists of a tuple labeling strategy and an adaptive retransmission strategy.The tuple tagging strategy adds the time tag to the root tuple ID by reconstructing the root tuple ID,which provides a tuple time monitoring method with low energy consumption and low latency.The tuple adaptive retransmission strategy calculates the timeout retransmission time to adapt to the current topology by collecting the root tuple time samples,and updates it to the topology configuration,so as to reduce the probability of overload of cluster components.Three groups of experiments are designed for tuple tracking strategy based on load balancing to verify the impact of the strategy on throughput,memory overhead and cluster load.The experimental results show that the tuple tracking strategy based on load balancing is 19% less than the default tracking strategy,and the number of tuples in the former is more stable than that in the latter,and the thread overload probability is greatly reduced.Three groups of control experiments are designed for the topology dynamic retransmission strategy.The experiment mainly tests the performance optimization of the algorithm in normal and extreme cases and in different topologies.When using the topology dynamic retransmission strategy,the performance and throughput of the cluster are improved,and the performance of the Sentence Word Count topology is the most obvious.Compared with the default retransmission strategy,the tuple failure rate and CPU utilization rate of the cluster are reduced by 16% and 0.6 GHz on average,and the cluster throughput is increased by 13% on average.Therefore,the topology dynamic retransmission strategy can effectively improve the performance of the cluster on the basis of increasing a small amount of time overhead. |
