Research And Implementation Of Collaborative Filtering Recommendation Algorithm Based On Spark Platform

The development of Internet technology has promoted the popularization of information technology in social life.All kinds of information data continue to grow,and it is an urgent problem to grab effective information from these huge amounts of data.One of the effective ways to solve this problem is to use recommendation system to analyze user's favorite information,create corresponding model and provide personalized recommendation system for users.However,the algorithms used in recommender systems still have imprecise problems.For example,when the number of users and items is increased,the recommendation accuracy of the algorithm will be lower.This paper mainly studies collaborative filtering recommendation algorithm,analyzes the advantages and disadvantages of various algorithms,and uses the items and Slope One algorithm to predict the score.First,the Slope One algorithm is used to predict the score and fill the scoring matrix.The advantage is that it can solve the problem of data sparseness,and on the other hand,the score prediction results can be used as one of the final results.Then,a new data set is formed by combining the original data set and the prediction data set.Calculate the similarity between items according to the preference matrix.Then,the similarity between items is used as the weight of Slope One algorithm to predict the score.Finally,training parameters,and the final results are obtained by combining the two parts with different ratios.In order to further improve the ability of the algorithm to process large amounts of data,this paper implements a Slope One similarity algorithm based on Spark platform and an improved collaborative filtering recommendation algorithm.Moreover,in order to compare the accuracy of the algorithm,this algorithm is implemented on the Hadoop platform.After repeated experiments on MovieLens datasets,the proposed collaborative filtering algorithm can effectively improve the accuracy of recommendation.And,compared to the Hadoop platform,the Spark platform is better suited for handling huge amounts of data..