Research And Implementation Of The Movie Recommender System Based On Hadoop

With the rapid rise of the Internet information technology,huge amounts of information spread in the network,and a complicated problem people are faced with is information overload,resulting in you cannot quickly locate the target information,search engines can help users retrieve information with keywords,but under the condition of too much retrieving results or user goals not clear,it is not an effective solution.Recommender system is currently regarded as a better solution,according to the history of user behavior it can intelligently dig the interest mode,and calculate targeted content.Now Recommender system has been successfully applied in many big Internet companies,such as e-commerces,video website and so on.The paper deeply studied the open source software distributed computing framework Hadoop,using the collaborative filtering algorithm with MapReduce programming model and distributed file system.Finally we designed and implemented a movie recommender prototype system based on Hadoop,the main work are as follows:1.The traditional collaborative filtering algorithm only consults rating data to calculate the similarity value between users and items,not thinking of the influence of other factors on the similarity calculation.Besides rating data extremely sparse will also affect the calculation of similarity.Therefore when calculating the collaborative filtering recommender algorithms based on users behavior,this paper set up the model in accordance with the user identity attribute at first,and then cluster the users within the class using the traditional collaborative filtering algorithm.2.The item based collaborative filtering recommender algorithms introduced the concept of similarity category,considering the rating data and item category,at the same time bringing in the influence of common rating users on the calculation of similarity.At last,in the selection of users nearest neighbor set,use the threshold method with average similarity and threshold to determine the target user or item nearest neighbor set,avoiding the noise caused by the fixed value method.3.Combined with distributed system solutions Hadoop,achieve an improved recommender algorithm based on the MapReduce programming model and HDFS distributed file system.4.Research the deployment and application of a distributed environment,combined with the improved collaborative filtering algorithm and system requirements,designed and implemented a movie prototype recommender system based on Hadoop platform.Experimental results demonstrate that the improved collaborative filtering algorithm can significantly improve the accuracy,while for big data sets,the improved distributed recommender algorithm is higher in operational efficiency.