Design And Implementation Of A Text Recommender System Of Social Network Based On Latent Dirichlet Allocation

In recent years, the rapid development of Internet and mobile Internet technology has gradually led us into an era of information explosion. People receive hundreds and thousands of fragmented information from SNS(Social Network Service) and network media every day. The development of technology improves people’s frequency and speed of message receiving, but in the meantime, it also makes people hard to find the content they are interested in from the gradually overloaded amount of information. Recommendation system is essentially establishing correspondence between similar users and data according to users’ behaviors and characteristics, thus predicting their interest. This thesis aims to design and achieve an optimized text recommendation system based on the background of SNS network to solve the drawbacks in traditional recommendation system.This thesis firstly analyzed the application and successful cases of the common recommendation algorithms currently, and analyzed their advantages and disadvantages. The traditional recommendation algorithm usually faces problems like cold boot and data sparsity. In order to solve these two problems, this thesis designed the integral structure of the recommendation system used by combing the actual characteristics of this system requirements, and eliminated the problem of cold boot brought by the frequently updated text content to some extent. By introducing in the staged training process, it reduced the system’s computation complexity in operation and improved its timeliness.Secondly, this thesis did a demand analysis of this recommendation system, finished the architectural design of the system by combining the index of performance and expansibility, and designed and achieved a complete recommendation system based on LDA(Latent Dirichlet Allocation) algorithm on the basis of this design. The system includes three main parts which are task system, storage system and Restful API. The task system is the core service, including offline training task, offline recommendation task and near-real-time recommendation task. The storage system achieves the distributed storage structure based on Hadoop HDFS to provide data storage for high availability, and optimizes the distributed computation based on Map/Reduce. Restful API, which is the external interface of the system, processes all the input and output data. Besides, the integral architectural design of this system also includes the Key/Value-based storage server called Redis, which is often adopted in Internet architecture, for improving the performance.In the end, this thesis performed an evaluation of this recommendation system based on the dataset of my project, test and verified the availability of recommendation algorithm through experiment and data.