Application And Research Of Collaborative Filtering Algorithm In Recommendation System

With the rapid development of economy and society,various kinds of goods are abundant,people have more and more shopping choices.There is an information overload problem that plagues people's choices.There are two ways to solve the problem,one is the search engine,another is the recommendation system.Search engines require users to extract keywords,then it can help users search for items,but it is difficult to extract keywords in certain degree.The basic task of the recommendation system is to establish the connection between the user and the item,it can solve the user's difficult shopping choices and recommend the appropriate items to the appropriate users.From the user's point of view,it analyzes the user's historical shopping behavior,and recommend items that purchased by users,who have similar purchase behavior with the current user to the current user.From the perspective of the item,it should recommend similar items to the current user.The ultimate goal of the recommendation system is to realize the accurate personalized recommendation of real-time online and meet the shopping requirement of users.The first model is based on three models that include the neighborhood-based recommendation algorithm,a matrix decomposition-based recommendation algorithm,and a time-based recommendation algorithm.Through the scalability of the matrix decomposition model,the item-based global neighborhood and the user-based global neighborhood are added to the model,and an improved fusion model is proposed.The two models that only added global user neighborhoods or only added global item neighborhoods were also studied.Finally,the RMSE and MAE of the fusion model that integrated three-factor have been reduced both in the training set and the test set.The second model is based on the time model,adding the user's time that influence the user's rating of the item into the recommendation system.The experiment shows that the model is better than the original model in the training set and the test set.Then a new model that integrated the svdplusplus model considers the advantages of the two models.At final,the algorithm shows that the final model gets good results both in the training set and in the test set.The third model is the Spark implementation based on the user-based collaborative filtering algorithm.User-based collaborative filtering algorithm has a wide range ofapplications.Spark ML does not implement a user-based collaborative filtering algorithm.This model uses the Mapper Reducer programming concept and the Spark distributed parallel computing framework to achieve the user-based collaborative filtering algorithm.The final algorithm is consistent with the collaborative filtering algorithm that does not use Spark in accuracy and prediction.The collaborative filtering algorithm based on Spark proves to be rewritten is successful and fills the blank of Spark ML.