Design And Implementation Of A Spark Based Hybrid Film Recommendation System

With the continuous expansion of the application scope of big data,it is difficult for people to find items that meet their interests directly,which wastes more time and costs,and leads to "information overload".In order to solve this problem,many technologies have emerged,among which recommendation systems not only screen out more favorite items for users,but also make the system improve user stickiness and personalized marketing revenue,and promote the development of recommendation systems in various fields.This paper focuses on movie recommendation systems.In the development process of recommendation systems,cold start,data sparsity,user interest shift and other issues affect the recommendation quality,and single algorithm and single point computing can no longer meet the needs of today’s various platforms.In order to better implement movie recommendation,this paper designs and implements a Spark based hybrid movie recommendation system.The main work is as follows:(1)Hybrid recommendation strategy: In offline recommendation,for cold start problem,this paper first directly displays offline statistical results for users;To solve the problem of data sparsity,this paper applies matrix decomposition to offline recommendation,introduces modified cosine similarity to calculate the similarity between movies,improves the accuracy of item based collaborative filtering recommendation,and uses ALS algorithm to train the best model to achieve model-based recommendation.At the same time,experiments are conducted to verify the feasibility of the algorithm;In the real-time recommendation,for the cold start problem,TF-IDF is used to process the tags that users put on movies,and recommend and optimize them based on user generated content;For the problem of user interest shift,this paper first collects users’ real-time rating logs and offline calculated movie similarity matrix,and optimizes the real-time recommendation priority model.After Spark Streaming calculation,the latest real-time recommendation list is merged.Finally,this paper shows the real-time and offline recommendation results to user partitions in a hybrid recommendation mode.(2)System design and implementation: first,conduct demand analysis,then analyze and design the system architecture,and use Spark to realize the calculation functions such as collaborative filtering recommendation based on articles,collaborative filtering recommendation based on ALS model,real-time recommendation,recommendation based on user generated content and offline statistical recommendation;Data storage is realized through Mongo DB,Redis and other databases,and system tests verify that functions and performance meet expectations.Finally,a movie hybrid recommendation system based on Spark is designed and implemented.To sum up,after the design and verification of the proposed hybrid recommendation strategy,this paper applies it to the movie recommendation system,which not only realizes the needs of user functions and experiences,but also alleviates the problems of data sparsity,cold start,and user interest deviation of the recommendation system to a certain extent,which has a certain reference significance for the design and development of the movie recommendation system.