Research And Application Of Recommendation Algorithm Based On Spark Platform

After entering the data age,the amount of data stored on the Internet has increased exponentially.In the face of massive data,general search functions cannot meet all needs and cannot fully tap user interests.A recommendation system was born,which helps users find high-quality information of interest faster,and can provide more accurate content than search methods even when user needs are vague.The thesis focuses on local similarity,nearest neighbor selection,and its application in collaborative filtering recommendation.Firstly,the concept of local similarity is introduced based on the different preferences of users,and the LSWSO algorithm is designed to improve the data density.The algorithm starts from the label,establishes the clustering algorithm based on the label factor,and uses the forgetting function to solve the problem of user interest drift.Secondly,in order to reduce the data sparsity,the weighted Slope One algorithm is used to fill the matrix.To improve the clustering efficiency,the LSWSO algorithm implements a parallelization scheme on the Spark platform.Secondly,the parallelized LSWSO algorithm is combined with user-based collaborative filtering,and the LSWSO-User CF algorithm is designed.After using LSWSO to obtain a relatively dense score matrix,the algorithm combines user attributes to calculate the similarity and selects the nearest neighbour set,and then uses the nearest neighbour as reference object for predictive scoring and Top-N recommendation.In addition,the LSWSO-User CF algorithm also uses the characteristics of LSWSO to propose a cold start solution.At the same time,based on factors such as the consistency of the development environment,LSWSO-User CF also uses Spark for parallelism Calculation.Lastly,a relatively complete personalized music recommendation prototype system was developed,and the parallel LSWSO-User CF algorithm is applied to the recommendation module of the system to verify the practicability of the algorithm further.Using Movie Lens and Tag-Genome datasets to experiment on the Spark platform,the test results show that the parallelized LSWSO algorithm and the parallelized LSWSO-User CF algorithm have better performance in big data scenarios.