Research And Application Of Hybrid Recommendation Algorithm Based On Spark Technology

The rapid development of Internet technology has led to the era of big data.The explosive increase in data volume has made the problem of "information overload" increasingly serious.How to quickly and efficiently filter out useful information from excessive information has become a problem that puzzles people.The recommendation system came into being.The working principle of traditional recommendation system is to use the past data to discover the characteristics of different user interests and preferences,relying on this function to personalize recommendations for users.But when facing massive data,traditional recommendation algorithms need to consume a lot of time and may not meet user needs.In this situation,the implementation of distributed Spark technology can effectively solve this technical defect.In the application of traditional recommendation algorithms,it faces the practical problems of sparseness,cold start and poor scalability.This paper is supported by Spark technology,combined with the traditional collaborative filtering recommendation algorithm to improve it.Based on this,a Hybrid recommendation model is constructed and applied to the movie recommendation field,which makes the recommendation effect significantly improved.The main research contents of this paper are as follows:(1)This paper briefly introduces the recommendation algorithms and related theoretical knowledge of Spark technology.It focuses on the classification and recommendation principles of collaborative filtering recommendation algorithms and their corresponding characteristics,then describes the Spark core technology and its working architecture.(2)Research and improvement of neighborhood-based collaborative filtering recommendation algorithm.Aiming at the similarity calculation based on the singularity of the scoring information,an improved similarity model that incorporates the concepts of local similarity and global similarity of user characteristics is proposed,then the user-based collaborative filter algorithm is improved.For the item-based collaborative filter algorithm,the similarity of label association is introduced,which is combined with the similarity between item ratings,so as to screen similar items more accurately,and thus improve the accuracy of recommendation.Finally,experiments are designed on Spark platform and the feasibility of the algorithm is verified.The experimental results show that the improved algorithm effectively improves the accuracy and scalability of the recommendation.(3)Collaborative filtering recommendation algorithm based on matrix factorization.Aiming at the sparseness of the scoring matrix,the matrix is decomposed by the alternating least square method,and the characteristics of the iterative decomposition are used to realize parallelizationbased on Spark technology.The group experiment is designed to verify the effect of different parameter values on the recommendation,and to obtain the optimal parameter combination of the model.(4)Hybrid recommendation model.Convert the general weighted mixture into an optimal solution problem,build a hybrid model based on improved neighborhood collaborative filtering algorithm and alternating least square method,and dynamically assign weights to the model by solving the optimal solution set.Design experiments to analyze the performance indexes of prediction accuracy and classification accuracy to verify the recommendation effect of the hybrid model.This model can be applied to the field of movie recommendation.