Research And Implementation Of Personalized Recommendation System Based On Streaming Computing

At present,the recommendation system occupies a decisive position in both e-commerce and social networks.Although the traditional personalized recommendation system can provide users with accurate recommendation services,there are still some defects in the face of changing user needs.Since the traditional system periodically calculates data in batches,the calculation results of the system cannot be fed back to the user in real time,and the recommended data cannot be updated in real time,which makes the recommendation lag.This is the real-time problem in the recommendation system.In addition,in the initial stage of the website,due to the lack of user-item rating data,the system cannot make effective recommendations,which is the cold start problem in the recommendation algorithm.In response to the above problems,this paper designs and implements a personalized recommendation system based on streaming computing.The improved recommendation algorithm that solves the cold start problem will be applied to the recommendation system,and the recommendation results will be updated according to the real-time data flow.This article mainly includes the following research contents:(1)Aiming at the cold start problem of the recommendation algorithm,a feature mapping algorithm based on similarity weighted KNN is proposed,named FM-SWK.The algorithm first uses the related similarity as the similarity measure of the KNN algorithm to find the nearest neighbor of the new user or new item,and then combines the attribute vector of the user or item to establish the new user or new item and the user with the existing score.The mapping relationship between items finally uses the feature information of a nearest neighbor user or item to estimate the feature value of the new user or new item through a similarity weighting method.Secondly,combining the FM-SWK algorithm and the matrix decomposition algorithm based on ALS-WR,a hybrid recommendation algorithm based on FM-SWK-ALS is proposed,which uses the FM-SWK algorithm to obtain the characteristic values of new users and new items,Recommend new users or new items through matrix decomposition model based on ALS-WR.At the same time,in view of the problem that the FM-SWK-ALS-based hybrid recommendation algorithm has high time complexity and irrelevant user and item feature vectors are also calculated,from the user and item aspects,an improved solution suitable for real-time data flow is proposed.The scheme incrementally calculates the feature vectors of new users or new items,avoids solving the feature vectors of unnecessary users or items,and then applies them to the above-mentioned hybrid recommendation algorithm.We use the public Movie Lens data set on the built Spark cluster to analyze the hybrid recommendation algorithm and its improvement scheme proposed above,and will use the matrix decomposition algorithm of ALS-WR and use the average value to fill in the missing scores of new users and new items.As a comparison algorithm,AVG-ALS is analyzed from both users and items.The value of MAE and RMSE based on the FM-SWK-ALS hybrid recommendation algorithm is significantly better than the two comparison algorithms,and its improvement schemes continue with the amount of data.Increase,iterative calculation time changes slowly,to meet the system calculation time requirements.The experimental results show that the hybrid recommendation algorithm based on FM-SWK-ALS can effectively solve the problem of being unable to effectively recommend due to the lack of user-item rating data to a certain extent,and improve the recommendation accuracy of the recommendation system.Effectively improve the recommendation efficiency of the algorithm.(2)In view of the problem that the traditional recommendation system has a slow calculation and cannot make recommendations based on the user's real-time behavior,this paper designs and implements a streaming processing architecture that can be calculated in real time.The architecture and the hybrid recommendation algorithm proposed in(1)are combined to build a personalized recommendation system based on streaming computing,and the system modules are designed and implemented according to demand analysis.They are mainly divided into real-time data flow modules based on Flume and Kafka,Real-time streaming computing module based on Spark Streaming and recommendation engine module based on MLlib in Spark,and using HDFS and Hbase for data storage,the first two are online processing part,the latter two are offline processing part.The system combines offline processing with online processing,that is,the offline processing part completes the calculation with high complexity and large amount of calculation,while the online processing part completes the lightweight calculation,so that the recommendation system can respond to user behavior in real time.This article tests the accuracy and real-time performance of the designed real-time recommendation system through experiments.Among them,the improved FM-SWK-ALS hybrid recommendation algorithm has an accuracy rate and recall rate that are 1.5% higher than the traditional User CF and Item CF on average;The real-time recommendation system built in this paper when the instantaneous number is within 8000 or the scoring rate is within 2500,the average response time of the system is within 5s.The experimental results show that the personalized recommendation system based on streaming computing in this paper has good performance.It can meet the needs of real-time recommendation while solving the cold start problem.