Research And Implementation Of Collaborative Filtering Recommendation System Based On Spark Large Data Processing

The information in modern society is more and more complicated.People live in the age of information overload.In some way,filtering useless information for users is the goal pursued by researchers.Recommendation system is a system to meet users' needs or services.It can meet users' personalized needs and occupy an important position in users' access and query information.However,there are still a series of problems to be solved in the development of recommender system,such as frequent business adjustment,slow response of the system,low accuracy of recommendation results and slow processing and analysis of massive data.In order to solve the above problems,it is necessary to improve the research of the recommended system.A good personalized recommendation system needs to have better expansibility on the one hand,and can adjust and update the system along with the continuous change of business needs.On the other hand,it also needs the technology of big data processing to solve the efficiency problem of recommendation process.At present,Hadoop and Spark distributed processing platform is an important solution to solve big data processing.It realizes massive data management and analysis through distributed computing and processing.This research is based on this analysis.The main problem that the recommendation system faces now is the sparsity of the data,which is also an important reason for the inaccuracy of the results of the recommendation system.With the increasing amount of data,the process of data processing has become complex.In order to solve the above problems more effectively,this paper designs and completes a recommendation system based on the Spark framework.In this paper,the overall structure and specific algorithm of the system are introduced,and the advantages and disadvantages of different algorithms are analyzed.Finally,a personalized recommendation method based on user preferences is finally selected.This paper mainly describes the research background and research status,and fully analyzes the main contents of this paper;then analyzes the relevant situation of the collaborative filtering algorithm and Spark data processing framework;topic similarity and semantic transfer respectively from the two aspects of algorithm to improve the recommendation in order to achieve the real-time effect;recommended,recommended lifting speed,using the Spark dataprocessing technique;after analyzing the design related algorithm,the design from the overall framework of the system,the overall framework of recommendation and recommendation engine engine design three parts to analysis design and realization;in order to prove the effectiveness of the algorithm,this paper made a lot of experiments and test.Finally,a movie website using this algorithm is realized.Experimental results show that the collaborative filtering algorithm in this paper has good effect of the recommendation,because of the use of Spark data processing framework,the speed advantage is obvious,in addition the application recommendation engine this topic in the movie website has achieved some success,has a certain theoretical value and practical significance of the study.