Connected Component Computing Technology Based On Large Scale Mixed Traffic Data

In this paper,the efficiency bottleneck of large-scale mixed traffic data connectivity component calculation is studied,and three problems are mainly solved:one is that the distributed graph computing algorithm represented by Google's pregel algorithm can't process the mixed traffic data of IPv4 and IPv6;the other is that only part of the traffic data can be collected when data collection in some fields,it takes more than a few days to process the connection component calculation between ten thousand level and ten million level sets,which can't meet the timeliness requirements;thirdly,there is a large amount of incremental connectivity calculation demand,a large-scale hybrid algorithm is developed The research of connected component computing technology of traffic data focuses on solving the above three problems,improving the efficiency of connected component computing of large-scale mixed traffic data,and the main work around this technology is as follows:First,we propose and implement an optimization algorithm for computing the connectivity component of the incomplete mixed traffic,to solve the problem of the processing of the mixed traffic of IPv4 and IPv6 and the calculation efficiency of the abnormal traffic.Secondly,we propose and implement the optimization algorithm of incremental connected component computing for off-line computing,make full use of the mapping relationship between IP address and connected component,replace the IP address that has been connected component computing with its connected component ID,and solve the problem that the existing connected component computing technology chooses to recalculate the full amount of data when facing new traffic data,which is inefficient Next question.Thirdly,based on this,we design and implement a connected component computing system based on large-scale mixed traffic data,and design experiments to verify the research results of this paper.Experiments show that the efficiency of the algorithm is improved by more than 23 times;the efficiency of the algorithm is improved about 70%in the calculation of incremental connected components for offline computing.