Research On Community Discovery And Opinion Leader Mining Method Based On Web Crawler

With the advent of the Internet era,the way people complete information exchange is no longer limited to real media.Many people are beginning to use the Internet for information exchange.Online social platforms are one of them.As one of the network information exchange platforms,Weibo is loved by many people.After more and more users use Weibo,a large number of user clusters make Weibo have the characteristics of social networks.Through the attention relationship and forwarding function between users,there is a close connection between users.Become a research hotspot.In this paper,the research on social network analysis of Weibo mainly includes the following aspects:1.Discuss the research status of the main research content of this article,and discuss related theoretical concepts and technologies.2.Obtain the resources in the social network platform through the distributed crawler system,and use Weibo as the goal to build a complete distributed crawler system,analyze and study the anti-reptile technology and solutions,and complete the analysis,processing and storage of data.Provide a data basis for the next two or three steps.3.Further analyze the microblog data obtained through the distributed crawler in step 1,divide the crawled microblog users into communities,and implement the parallel Louvain algorithm based on GraphX through the Spark parallelization framework,which makes the algorithm run efficiently Promote.4.Studyed and studied the community opinion leader discovery algorithm,based on the idea of PageRank algorithm,comparatively analyzed the similarity between the community network and the web link network,and found the user's initial influence characteristics and user interaction behavior influence in the community network Characteristics,and design and implement the core member discovery algorithm of WeiboLeaderRank community according to these characteristics.