Design And Implementation Of Movie Recommendation System Based On Spark

As an important element of entertainment,movies meet people's needs in life.But with the passage of time,film resources have gradually expanded,making it difficult for people to find out which films they are interested in.For this kind of information overload problem,the recommendation system is a good solution.The recommendation system digs out the user's preferences by analyzing user,item,context and other information,and actively recommends items that satisfy the user's preferences to the user.At present,the theory of recommendation system is relatively mature and its application is very extensive,but different recommendation algorithms have their own defects.In order to cope with the deficiencies of a single algorithm,combining multiple different algorithms with a combination strategy can significantly improve the accuracy of recommendation.This article first analyzes the functional and non-functional requirements of the movie recommendation system.In terms of functional requirements,the system needs to have basic user behavior functions and movie recommendation functions.In terms of non-functional requirements,the system needs to meet performance indicators such as real-time and accuracy.Then,according to the development software framework and business content of the system,the software architecture system is designed,and it is divided into the view layer,the business layer,the recommendation engine layer and the data storage layer.The recommendation engine layer is the core layer,including four recommendation scenarios:an offline recommendation module based on the GBDT and LR fusion model,a real-time recommendation module based on Spark Streaming,a popular recommendation module based on Spark SQL,and a detail page recommendation module based on ALS.Among them,the offline recommendation module uses word vector embedding to expand the features of the recommendation model to improve the accuracy of the model.The real-time recommendation module uses Spark Streaming to ensure the real-time performance of movie recommendation.It also made a detailed design around the recommendation engine,using the data preprocessing module to speed up the recommendation calculation,using the log module to update the recommendation model and recommendation results,and using the movie vectorization module to expand the model features.Then,the movie recommendation system is implemented,which mainly describes the implementation of the data preprocessing module,log module,movie vectorization module and four recommendation modules.In the process of realization,by designing different classes and methods,data extraction and various transformations are realized.Finally,the experimental data set is explained,the system test environment is deployed,the functions of the movie recommendation system are tested,and the performance test is carried out.For performance testing,the accuracy of the offline recommendation model incorporating movie word vector features has been significantly improved,and the real-time recommendation module can also achieve a second-level response.