The Design And Implementation Of Distributed Recommender System Based On Hadoop Platform

As the widespread applications of internet technologies, the increase of information and the decrease of the cost to get information lead to an information overload problem. As one of methods to filter information, a recommender system can be applied in some scenrios where the items are difficult to be described by attribute features. Besides it can give the items in long tail more opportunities to show as well. On the other hand, the sharp-increase of data is becoming a challenge for traditional web applications deployed on one server. So the recommender system should be improved in architecture design to resolve or mitigate the data size problem.Therefore, to improve the accuracy rate of recommender algorithms and to increase the scalability of recommender systems are the main topics of this thesis. In order to simulate and forecast the user behavior more precisely, the MovieLens dataset has been analysed, and the common factor and the specific factor are abstracted to described the user behaviors as the conclusion. Based on the conclusion and analysis of the classic recommender algorithms on these two dimensions, two recommender algorithms optimized for these two factors have been proposed. To simulate individual user behavior more precisely, the two algorithms optimized for the two factors are used to build a hybrid recommender system by weighting their results on a scale learned from every user’s behavior model. In the consideration of scalability of recommender systems, the hybrid recommender algorithm is reprogrammed base on the MapReduce programming model to make the recommender tasks can run on the Hadoop. In the end, a design of a recommender system including online and offline system is proposed and a case of the recommender system implementation is presented.In this thesis, the hybrid recommender system presented can simulate the user behaviour more precisely. The system proved to have higher accuracy rate and flexibility. At the same time, the implementation based on Hadoop make it convenient to extend while the dataset is increasing.