Design And Implementation Of Recommendation System Based On Hadoop

With the increasing development of Internet technology,the amount of data is increasing with each passing day.The recommendation system has become the first way to help users filter effective information in massive data.However,in actual application process,due to data sparseness,cold start,and high repetition rate of recommended content,resulting in a poor system experience.And with the advent of the era of big data,the extraction,analysis,calculation,and storage of data are also issues that need to be addressed in the design and implementation of recommendation systems.In response to these problems,this article first summarizes and learns the background of the recommendation system and its algorithm and the Hadoop distributed framework,as well as the development status at home and abroad.And then researches and analyzes the recommendation algorithm based on collaborative filtering.Combining the solution of the similarity in the algorithm with the network structure,a collaborative filtering recommendation algorithm based on user / item neighborhoods is proposed.At the same time,combined with numerical experiments,the improvement effect of the algorithm is verified.Finally,the architecture requirement analysis and function design and implementation of the movie recommendation system are completed to achieve the development of a personalized movie recommendation system.The main tasks are as follows:1.Based on the current Internet situation,the users and items in the recommendation system are abstracted into a network structure model.And based on the bipartite structure,the user-level network and the item-level network are considered the influence of solving the similarity in the collaborative filtering recommendation algorithm.We proposed an improved algorithm for weighted similarity,and it is applied to collaborative filtering recommendation based on user neighborhood and collaborative filtering recommendation based on item neighborhood.At the same time,numerical experiments are carried out in the corresponding data sets,considering two scenarios of user binary scores and multivariate scores respectively,and the performance of the improved algorithm is verified by evaluation indicators such as root mean square error,accuracy,and recall rate of prediction results.2.Based on the improved algorithm,combined with big data Hadoop and Spark framework,the design and implementation of a personalized movie recommendation system platform,mainly including the basic functions and recommendation functions of the system.The basic functions involve the back-end and front-end of the system.At the same time,the system can obtain the user's behavior data and complete the pre-processing functions such as data deduplication,word segmentation,and extraction.The core recommendation engine part adopts improved collaborative filtering recommendation algorithm and content-based recommendation algorithm respectively,and designs offline recommendation service,real-time recommendation service,statistical recommendation service and content-based recommendation service,etc.to realize personalized movie recommendation for users Satisfy the user's viewing.