Design And Implementation Of Film Recommendation System Based On Hadoop Platform

With the rapid development of Internet and the emerging of big data comes along the issue of “data overload” where people needs to spend much more energy and time to search for desired information.Thus how to recommend the most useful and valuable information to users,as well as improving users quality of experience(QoE),has become a crucial research problem.Collaborative Filtering Recommendation System(CFRS)is a classical and proven useful system which can recommend desired information to users.With the dramatic increase of users and items,however,some huge drawbacks of CFRS,such as inefficient and lack of expansibility,have become apparent.To solve the above problems,this paper proposes an improved recommender system model named NE-UserCF(NMF-E2LSH-UserCF)based on Non-negative Matrix Factorization algorithm(NMF)and Exact Euclidean Locality Sensitive Hashing algorithm(E2LSH)to improve the quality and performance of recommendation.Then validate its validity and reliability by leveraging the MovieLens dataset.Finally,a complete and integral recommendation system based on NE-UserCF model is designed,and further implement it with the Hadoop.The main work and contributions of this paper are as follows:(1)Designing the NE-UserCF model: NMF can be used to eliminate invalid and redundant information in user-rating matrix(URM)whilst keeping the non-negative characteristic of the factorized matrix.Preprocess the URM with NMF,eliminating invalid and redundant features,to reduce the dimension of URM,accelerate the similar users search process,and improve the quality of recommendation.Then by taking the advantage of E2 LSH that can guarantee the result's accuracy and integrity,use E2 LSH to construct users' indexes,cluster similar-interest users(SIUs)and get the similar-interest-user matrix(SIUM).Wefurther process the Top-N recommendations by adopting the User-based Collaborative Filtering algorithm(UserCF).(2)Verifying the reliability of NE-UserCF model: Implement it with MapReduce,then use MovieLens dataset to conduct the off-line batch training to validate its reliability.Experiments indicate that it can improve the quality and accuracy of recommendation.(3)Implementing a Hadoop-based recommendation system integrated with NE-UserCF Model: The main components of this system include personal information management,movie information management,recommendation results retrieval,and recommendation results management.The Hadoop-based recommendation system which is integrated with NE-UserCF Model is finally implemented after a series of processes including setting up Hadoop,implementing recommendation system and deploying related algorithms.It achieves the expected quality of recommendation.