Research And Implementation Of Collaborative Filtering Recommendation System Based On Spark Big Data Processing

Recommendation system is used to recommend users to meet their needs or services,which can serve as a link between users and the information,to provide users with practical information.However,in the process of recommendation system development,there are also a series of difficult problems to solve,such as the speed of system response,the accuracy of recommendation results and the processing and analysis of mass data.In order to solve the above problems,it is necessary to continuously research and upgrade the recommendation system.The recommendation system needs to ensure better scalability,and can adjust and update the system as the business needs change.In addition,big data processing technology is needed to solve the efficiency of the recommendation process.Hadoop and Spark distributed processing platform is an important solution to large data processing,and it can manage and analyze massive data through distributed computing and processing.The recommendation algorithm is the core of the recommendation system.The biggest problem faced by the recommendation algorithm is the data sparsity of the user-item rating matrix.In practical applications,as the number of users and projects increases,the sparseness of the user-item scoring matrix tends to exceed 97%,and data sparsity becomes the biggest problem affecting the recommendation accuracy.Therefore,certain methods must be taken to reduce effect of it.In addition,as the amount of data continues to increase,the processing of data becomes more complex.In order to solve the above problems,a collaborative filtering algorithm based on project similarity transfer is proposed in this paper to solve the problem of data sparseness.Combined with the improved implicit semantic calculation model,the accuracy of recommendation results is improved,and the algorithm is proved by experiments.This paper first describes the research background and current situation of subject,then analyzes the relevant circumstances of the collaborative filtering algorithm,and the related technology of Spark data processing framework.The main task of this paper is to improve the recommendation effect from similarity transmission and semantic analysis algorithm.After analyzing the related algorithms,the paper analyzes and designs the system from three parts: the overall framework of the system,the overall framework of the recommendation engine and recommendation engine design.In order to prove the effectiveness of the algorithm,this paper tests and test process of a large number of.Then,a movie website using this algorithm is implemented.Facts proved that the system filtering recommendation algorithm based on Spark large data processing has achieved great success in the movie website.This research has good application value.