Design And Implementation Of Distributed Recommendation System Based On Spark

The world is in an era of information technology.The total amount of global information is increasing rapidly,but its value is also growing.The recommendation system can mine information required by users from the massive data according to the basic information and historical behaviors of users and provide personalized recommendation services to users or articles automatically.Nowadays,collaborative filtering is one of the most successful and popular recommendation methods in the whole recommendation system.The research on the collaborative filtering recommendation algorithm has been on the rise,but there are still many problems to be solved.The traditional recommendation algorithm implementation process requires a large amount of computation time,long time delay,poor timeliness,and can no longer meet the current business requirements.At the same time,most of the recommended algorithms now have data sparseness and cold start problems,which seriously affect the accuracy of the recommended results.Therefore,in view of the above status quo,the subject will mainly start from the timeliness and accuracy of the recommended algorithm.(1)The timeliness of the recommendation system is optimized for the platform used by the recommendation system.The platform adopted by the recommendation system is Spark,which has superior performance in data complex processing,analysis,and computation iterations.Parallelization of the als model recommendation algorithm based on this platform is completed to increase the data processing speed;thus,the recommendation system takes more time.Short,user experience is better.(2)Recommend the accuracy of the system and optimize the implementation process of the recommended algorithm.The proposed algorithm based on the ALS model is implemented in parallel on the spark platform.Considering that the lack of accuracy of recommender systems based on the ALS model,the absence of similarities in items,and user interest oblivion with time migration,The subject compares several common similarity calculation,and incorporates the appropriate item similarity calculation into the loss function to reduce the loss of attribute information of the invisible factor item,and introduces the interest forgetting function into the prediction score so as to achieve real-time accuracy High recommended.The subject uses the publicly available MovieLens dataset.Comparing the experimental results,it is found that the timeliness and accuracy of the recommendation system are effectively improved by optimizing the platform and algorithm used in the recommendation system.