Research And Implementation On Community Detection Based On Latent Dirichlet Allocation

With the rapid development of Internet technology in recent years,social network services attracts more and more attention.In the real social relations,we can build a very complex social relations network by taking a user as a node and link between users as an edge.Human social and geographical,cultural and other factors also determine that the social relations network must be composed by social communities with large or small scale.Community detection is very important for understanding the structure of online social networks and its evolution law,which is very important for understanding the depth of social relationship and human behavior.From the perspective of application,in social networks,the virtual community can be found in the Internet service providers to find the user's behavior patterns and interests,so as to provide users with personalized services.And in the level of information security,virtual community detection technology in dealing with fake users and controlling of illegal information dissemination has gained very high practical significance.This paper analyzes the existing theories and technologies in the field of community detection,sums up the problems it faces and then proposes a LDA-based model in considering both model interpretability,computational efficiency and the form of real datasets from SNS vendor.The main work of this paper and the research results include:1.This paper introduces background of social network analysis,and related works of community detection,and then summarizes and analyzes the existing problems and put forward the corresponding solutions;2.Based on the Bayesian graph model,this paper proposes a community detection model COT(Over Time Community),by considering both the interaction information,network structure and the timestamps of interaction behaviors,COT can be used to find dynamic communities in the interactive data from the online social networks;3.With the requirements of big data processing,this paper also presents a distributed version of COT model AD-COT(Distributed COT COT),which can be used in parallel and distributed computing environment,and greatly improves the scalability of COT model;4.This paper implements COT and AD-COT model on the Apache Spark platform,experiments are then carried out to verifying.And in the end,we analyzes the results and summarizes the future directions of our works.