Research On Combined Recommendation Algorithm Based On Spark Platform

In the context of the big data era,the contradiction between the limited ability of Internet users' attention and the infinity of massive data is increasing.The recommendation system is an indispensable tool to alleviate the contradiction.Collaborative filtering algorithm is one of the recommended algorithms commonly used in current recommended systems.Although it has many advantages such as speed of recommendation and recommendation effect,there are some difficult problems.For instance,storage and computing efficiency degradation issues will appeared when deal with a huge amount of data.that will also lead to the lack of timeliness issues.Or when the amount of data is scarce and the ratio of new users and items is too high,it will lead to the issues of low quality of recommendation and deviation of prediction results.In view of the rapid development trend of multifarious applications based on the Spark framework around the world,we mainly study the relevant recommendation algorithm of using MovieLens dataset by building a Spark cluster environment based on HDFS(Hadoop Distributed File System)and YARN(Yet Another Resource Negotiator)in this paper.The main work contents are as follows:(1)In our study,we optimize the method based on ALS(Alternate Least Squares)model and select a better parameter value by a dynamic verification method.after obtaining the number of eigenvalues and iterations of the recommended environment,the algorithm can more accurately predict the corresponding movie score.(2)We combine the scoring scale and average calculation with cosine similarity algorithm in our study to improve the Item-based collaborative filtering algorithm and User-based collaborative filtering algorithm respectively.Through the experiment on Spark platform,the comparison shows that the improved effect is better,which solve the sparseness problems that occur in traditional algorithms to some extent.(3)This paper proposes a combination recommendation algorithm combining similarity algorithm and ALS algorithm based on Spark platform.It uses the idea of weighted composition to combine "collaborative filtering based on item similarity," "collaborative filtering based on user similarity," and "collaborative filtering by alternating least squares(ALS)".The combination of building a dynamic weighted regression model,which more advanced than traditional linear weighting.using the model of loss function optimization calculation method,the effect of the improved algorithm combination.The experiment results show that the new model has a better prediction accuracy under the objective function,which effectively relieves the cold start problem caused by sparse data and improves the accuracy of the systematic recommendation.In short summary,our improved algorithm is more accurate than the traditional one.And the advantage is more obvious when scoring sparse matrix.