Research And Implementation Of Louvain Algorithm In Community Mining

With the rapid development of modern network technology such as Internet and cloud computing,we are enjoying the convenience they bring to our life.People have gradually entered the era of big data,exponentially huge amounts of data not only bring challenges and opportunities for the emerging Internet enterprise,but also for the traditional industry.The complexity of the network not only tests the existing IT infrastructure,but also challenge the computer's calculation ability.In the age of the social networking platform,the major social networking sites produce terabytes of data every day.Traditional stand-alone operation is difficult to meet the processing of large data,with the development of distributed computing framework,and the mature of parallel computing technology,it has been able to effectively solve the bottleneck of calculation and provide technical support for massive amounts of data mining.From the perspective of the social network research,the key of the research for social network is the relationship and social attribute between the nodes.The reality shows that real social network has the common characteristics of community structure,and the process of mining community structure through the network and the connection of nodes is called community detection.Louvain algorithm is an efficient algorithm based on optimization module degrees,in addition to the advantage of time,it can also detect the hierarchical community structure,missing none of the small community.On the basis of the Louvain algorithm study,in order to improve the effectiveness of the algorithm,we supply a new idea that can work parallel for Louvain and give a sample of how to implement it,at the same time guarantee the precision of the algorithm remains the same.The development of distributed computing framework has promoted the application in distributed computing,but community detection algorithm in the distributed computing has little correlation.In the face of huge amounts of data in large-scale complex networks,the application of distributed computing framework will further improve the operational efficiency of the algorithm,the combination of community discovery algorithm and distributed computing framework must be the hot spot of the research in the future.The advantage of Spark relative to the Hadoop distributed computing framework lies in its cache,it can save much disk access time,and it has a good ability to adapt the community detection algorithm such as which needs a large number of iteration algorithm.This paper implement the improved algorithm on Spark distributed platform,further improve the operation efficiency.In order to save the problem of community swap due to the messaging delay between different computational nodes in the distributed computing,this paper put forward a connected graph method to solve this problem.