Research Of Mean Model For Collaborative Filtering Recommendation Algorithm

Nowadays, the rapid development of the Internet has promoted ecommerce development. With an increasing of transaction data, the most effective Hadoop, spark and other data processing technology still exist many problems. For example, some data mining algorithms with better effects in the evaluation sets, which is hard to get ideal effect for real transaction data or difficult to achieve. It becomes an urgent task to explore a big data processing method for real applications.Firstly, this thesis introduces the basic principle and concept of recommendation system, and analyzes item based collaborative filtering(IBCF) algorithm in detail. Additionally, we describe the basic principle of the Mean Model for data compression, and the collaborative filtering recommendation algorithm based on the model is studied systematically. At last, it turns out that the operation effect of IBCF algorithm based on Mean Model in MapReduce framework valid to some extent. The main contents include(1) Evaluation research of collaborative filtering recommendation algorithm:After analyzing the collaborative filtering recommendation algorithm, it can be found that computing tasks mainly focus on the item similarity calculation. Meanwhile, evaluate all various of different similarity calculation methods, and point out that the cosine similarity has the best accuracy. And then the different evaluation indexes are evaluated, and each evaluation value is analyzed.(2) Mean Model and its improvementBased on the preliminary exploration of Mean Model, the basic principle and properties of Mean Model are analyzed and summarized. It points out that original model exists disadvantage of hierarchical fuzzy and information distortion. Then, an improved model is proposed, the IBCF algorithm based on Improved Mean Model reflects hierarchical idea, and overcome information distortion, which has better effect.(3) Research the incremental extension of Mean ModelData in the Internet grows every minute, all kinds of data applications must integrate into the new data to ensure the quality of the service system. For example, the recommended system will have a large number of new data generated every day; it must be operated in time, thus ensuring the quality of the recommendation. This thesis exploits IBCF algorithm as application background, in view of the incremental update of the Mean Model, an implementation method of Incremental Mean Model(Incremental MM) is proposed. Incremental MM can establish a statistical mapping table through the pre-statistics of item score, effectively support the incremental conversion of the Mean Model. The experimental results on Movie Lens dataset show that IBCF based on Incremental MM has the higher updating efficiency, and avoid the loss of the recommendation accuracy.(4) Parallel implementation of IBCF algorithm based on Mean ModelIn order to evaluate application effects of Mean Model in big data processing, the IBCF algorithm based on Mean Model is designed, and experiments are carried out on the Hadoop cluster with Netflix data set. The experimental results show that the Mean Model can achieve good effects in the process of big data processing.