Design And Implementation Of Distributed Personalized Recommendation System

With the explosive growth of mobile Internet information and the continuous advancement of machine learning technology,the use of recommended applications based on big data is becoming more widespread.However,the source of information for most recommended applications is limited to the information source set by the application developer.The user cannot select the information source and limits the diversity of the information content.The system supports user-customized information sources of interest,making it easier and faster for users to obtain information of interest.Simultaneously,the system is also provided to the users with real-time hotspot word cloud display,information search and personalized recommendation service based on big data,which makes the difficulty of users obtaining information of interest further reduced.The paper is based on the current popular big data technology,using Spark big data computing framework to analyze and calculate the information data obtained from the network,constructing a user behavior log collection and analysis system,and modeling user behavior.Personalized recommendations based on information modeling data and user behavior modeling data.The project integrates various technologies,and the system provides users with information classification browsing,information search,and information personalized recommendation services.Information classification browsing uses the Scrapy-Redis distributed crawler to obtain information data on the Internet.Organize the data into structured information and write it to the HBase database to create an information source library.The TF-IDF algorithm is used to extract the information subject words and combine with WordCould to generate real-time hotspot word cloud.The information search is based on the distributed search engine Elasticsearch,which designs the search interface according to the user's information acquisition requirements.The information personalization recommendation calculates the interest value of the user to recommend the information according to the similarity between the topic distribution vector of the information and the user preference behavior vector.The topic distribution vector of the information is based on the LDA topic model to model the information in the information source library,and is calculated by the Spark distributed computing framework.The user preference behavior vector collects the user preference behavior log by using the webpage burying technology,and processes the user behavior log timely based on the stream processing technology and calculates it by the time weight attenuation function.Tests show that the system is stable and there are no obvious errors,and the design and implementation methods are feasible and effective.After the system is released to the test environment,the tester has received good feedback,indicating that the system can accurately recommend the information to the user,which greatly reduces the difficulty of information acquisition.