Research On Dynamic Community Discovery Based On Spark Streaming And Its Application In Personalized Recommendation

As an important technology of network analysis,community discovery can find out some common nodes in the network.The research of the community in the network plays an important role in understanding the structure and function of the whole network,it can not only help us to analyze and predict the interaction between the elements of the entire network,but also to analyze the behavior of users and to provide users with more personalized search results.In reality,community discovery has played an important role in many fields.This thesis proposed a Louvain algorithm based on leaf community and node degree comparison strategy,based on the in-depth study of community discovery algorithm,Spark related technology and personalized recommendation technology,and integrate improved Louvain algorithm into the Spark Streaming flow processing framework,whitch can adjust the community structure and capture the community information in real time.Finally,this thesis apply the idea of community discovery to the field of personalized recommendation,which is used to solve a large number of vector operations.The main work of this thesis is as follows:1)This thesis propose an improved Louvain algorithm,including the leaf community strategy and node degree comparison strategy.First,Leaf community refers to the community,which contains the leaf node and the total degree of community is 2n-1(n for the number of nodes).Second,the leaf community strategy refers to directly divide the nodes of the leaf community to neighborhood community,whose degree is less than or equal to 2.Node degree comparison strategy is to directly compare the value of the neighboring nodes of the sigma tot to find the neighbor node of Max Delta Q.The improved algorithm greatly reduces the computation of the Q value and improves the execution efficiency.2)Aiming at the problem that the Louvain community discovery algorithm is only suitable for the static social network community discovery,this thesis propose a dynamic community discovery framework based on Spark Streaming(SDCDF),the dynamic community discovery strategy in SDCDF reduces the number of communitypartition and improves the efficiency of community dynamic discovery.3)Aiming at the problem that the traditional movie recommendation model leads to the complexity of the vector operation and the matrix consumption system is too large with the increase of the number of users and the film,this thesis propose a movie recommendation model(LFRM)based on the improved Louvain algorithm.According to the result of Louvain improved algorithm,LFRM transform the matrix of User-Movie into a matrix of Community-Movie,then use the ALS training model and predict preference value and recommend movie.This model avoids a large number of vector operations through transforming the matrix of User-Movie into a matrix of Community-Movie,and improves the efficiency of personalized recommendation to a certain extent.