Design And Implementation Of Movie Recommendation System Based On Hadoop

With network technology and computer technology developing, the Internet has went in the era of Web2.0. The recommendation system has been widely applied to the field of music, social, video and microblogging sites and it is becoming the important filtering tools to solve the problem of "Information Overload" which can help users get the information they need in the massive data. However, in actual applications, the types of products and number of users and are generally very large and the conventional recommendation system tends to run in the environment of single machine. In the result, the performance was limited, the efficiency of analysis and calculation was reduced, the extended storage space was inadequate, and the user interaction was not strong and so on. It has been incapable to satisfy the demands of the Internet’s rapid development. The main content of this paper is how to efficiently deal with the recommended system with large data sets.To improve the efficiency of recommendation, solve the problem of scalability, this paper proposed a solution based on Hadoop that is the distributed open source framework, using the approach of parallel computing to improve recommendation algorithm. In the end, this paper designed and implemented the movie recommendation system based on Hadoop. The main contents include:(1) When producing the technologies, this paper firstly researched and analyzed the architecture, operation mechanism and programming principles of Hadoop from HDFS and MapReduce. Secondly, this paper brief analyzed the recommendation system and related recommendation algorithm. Finally, this paper described the open source project Mahout, laying a theoretical foundation for the follow-up work.(2) This paper minutely introduced the principle and computing process of the most common used recommendation algorithm: User-Based Collaborative Filtering (UB-CF) and Item-Based Collaborative Filtering (IB-CF). After comparing the two algorithms, this paper finally choose the IB-CF to study.(3) This paper proposed the improved solution of IB-CF based on MapReduce according to the problem that the efficiency of recommender was very low in the environment of single machine. This method just used four times of MapReduce tasks to complete the recommendation.(4) In the end, this paper designed and implemented the movie recommendation system based on Hadoop, and proved the method proposed in this paper having high efficiency comparing to the traditional method by the experiments.