Implementation And Optimization Of Real-Time Recommendation System Based On Spark Streaming

The recommendation system needs to be able to recommend personalized information to the user.However,today's recommendation systems usually do not recommend personalized information in real time and dynamically,which is a real-time problem with the recommendation system.In addition,there is often a problem that the user/item scoring matrix data is sparse.At the same time,since new users or new items do not have comment information,it is not possible to accurately recommend new users or new items,which is a cold start problem in the recommendation system.For the above questions,This paper proposes p-similarity to solve the problem of sparse data sparging by traditional collaborative filtering algorithms.Based on the p-similarity,this paper proposes a CF-Kmeans hybrid recommendation algorithm,optimizes the matrix-based collaborative filtering algorithm,and alleviates the cold start problem in the recommendation system.At the same time,this paper studies the real-time recommendation application and proposes an algorithm suitable for real-time recommendation.The main research contents of this paper include:(1)For the data sparse problem of the recommendation system,this paper proposes a similarity calculation method for p-similarity.The method optimizes the process of similarity calculation in the traditional collaborative filtering algorithm.The p-factor is added to the similarity of the collaborative filtering algorithm.The recommendation algorithm using p-similarity better adapts to the sparse data and alleviates the data.Sparse impact on recommendations.(2)For the cold start problem of today's collaborative filtering algorithm,a CF-K-means hybrid recommendation algorithm is proposed,which solves the cold start problem in the traditional collaborative filtering algorithm.The collaborative decomposition algorithm based on matrix decomposition decomposes the feature matrix of the user/item,and then clusters the decomposed new users/new items to obtain the K nearest neighbors of the new user/new item.Calculate the feature vector of the new user/new item using the feature information of the K nearest neighbors,and then supplement the comments in the score matrix that the new user/new item does not have,use the p-similarity calculation method to calculate the similarity,and perform the recommendation to solve the cold Startup issue.Based on the algorithm,an incremental hybrid recommendation algorithm is proposed to improve the recommendation efficiency of the algorithm in real time.(3)Research on real-time recommendation application based on Spark Streaming.This paper studies the real-time recommendation application and proposes the overall architecture of the recommendation system,which is divided into data display layer,data processing layer and data storage layer.At the same time,the key modules of real-time recommendation are researched and implemented,including four key modules: data simulation and subscription release module,flow calculation module,real-time recommendation system recommendation module,distributed storage module.The data storage of this paper mainly uses Hbase.achieve.Finally,the real-time recommendation system was tested to verify that the real-time recommendation system proposed in this paper meets the requirements.