Improvement And Implementation Of Slope One Collaborative Recommendation Algorithm Based On Spark

The rapid development of Internet technology has brought a lot of information.On the one hand,users want to retrieve the information they are interested in quickly.On the other hand,enterprises want to recommend their products accurately.Recommendation system is born under such circumstances.Among all kinds of recommendation algorithms,collaborative filtering algorithm is the most popular one,and Slope One algorithm is widely used for its easy implementation and high prediction accuracy.However,the traditional Slope One algorithm ignores the correlation between users and items and the similarity between items in the scoring prediction,so the effect of personalized recommendation is not good.In view of the above problems,this paper analyzes and discusses the collaborative filtering algorithm in detail,the specific work is as follows:(1)The flow and problems of Slope One algorithm are studied.In the process of recommendation,Slope One algorithm only considers the user's score deviation matrix.In this paper,an improved Slope One algorithm based on item similarity and user item correlation is proposed,which combines item similarity and user item correlation as the prediction weight to improve the prediction accuracy of the algorithm.(2)The traditional neighbor selection method is studied.Aiming at the problems of traditional k-value selection method and fixed threshold selection method,the similarity threshold is divided dynamically,and the average absolute error is verified by several algorithms.By observing the data,we can get the best similarity threshold,and then select the nearest neighbor,which can improve the prediction accuracy and reduce the amount of data operation.(3)The effect of the improved algorithm is verified based on Spark platform.Three experiments are designed: Firstly,in order to determine the appropriate threshold and training set,the Item-based Collaborative Filtering algorithm,Slope One algorithm and the improved Slope One algorithm are compared in three datasets.By comparing the MAE results under different thresholds and training set proportions,the appropriate threshold and training set ratio are obtained.Secondly,by adding the User-based Collaborative Filtering algorithm and the Weighted Slope One algorithm,the paper compares the three evaluation indexes of the five algorithms under three data sets: accuracy rate,recall rate and the value of average absolute error,and analyzes the performance of each recommendation algorithm under these indexes comprehensively.Finally,the paper runs the improved Slope One algorithm on the Spark platform and Hadoop platform respectively,and finds that the running time based on the Spark platform is shorter.In conclusion,the improved Slope One recommendation algorithm based on Spark platform proposed in this paper has high prediction accuracy and speed.