Research On ALS Collaborative Filtering Recommendation Algorithm On Spark Platform

With the rapid development of internet and internet of things technologies,the data generated by human beings have grown exponentially.The world every day will produce a lot of data,how to deal with big data quickly for human use that we urgently need to solve,along with the continuous development and maturity of big data and data mining technology,our data processing technology is also improving.The recommendation system is an application that predicts target users to recommend related products based on users' historical behaviors and peace-of-mind information.Nowadays,personalized recommendation technologies play an increasingly important role in our life and are widely applied E-commerce,news push,video music recommendation and so on.Among the many distributed computing frameworks,Spark's high level of fault tolerance,scalability,and ease-of-use make it a subject of great interest in recent years.Spark is now a relatively hot memory-based general-purpose parallel computing big data engine,because of its advantages in iterative parallelization,is widely used in big data processing.In this paper,the least squares algorithm(ALS)and the improved algorithm based on the matrix factorization of collaborative filtering are mainly studied and tested on the Spark platform.In this paper,firstly,the performance and logic of ALS algorithm are analyzed and studied.Through a detailed study of his characteristics,it is concluded that the parallelism function of Spark platform is very suitable for the operation of ALS algorithm.In order to get the optimal parameter model of ALS algorithm under our experimental conditions,we use the step-by-step approximation method to compare the performance parameters MSE,RMSE and MAE by multiple experiments to get the optimal parameter model of ALS algorithm.ALS algorithm-based recommendation system In this experimental environment,we use the parameters obtained through experiments,the prediction accuracy has been significantly improved.Then we analyze some problems existing in the ALS algorithm.A large number of iterative operations seem to be one of the many advantages of the ALS algorithm.Therefore,we improve a large number of iterations of the ALS algorithm.In this paper,Nonlinear Conjugate Gradient Algorithm(NCG)with Similar Convergence.By using the NCG algorithm to reduce the number of iterations of the ALS algorithm,we can further reduce the recommended time and improve the efficiency of our real-time recommendations.Finally,through analysis and experiment,we prove that the improved ALS algorithm incorporated into the NCG algorithm significantly reduces the number of iterations and improves the accuracy.Initially reached the recommended effect we want.Finally,we summarized the full text.