Research On An Implicit Feedback Recommendation Framework Utilizing Distributed Memory

The Internet has greatly changed the way we live. With the rapid popularization and development of information technology, all kinds of Internet services, whether it is e-commerce platform, social networking sites or online video sites, have collected vast amounts of data during their operations. How to deal with this information, dig out the potential of knowledge and make use of it to generate revenue has gradually become a commercial focus. Recommender system, as a user preference mining technology, have been studied extensively both in academia and the business community.Matrix factorization based methods have achieved better performance in the recommendation contest named Netflix Prize, and attracted a large number of researchers for the improvement and innovation. However, these methods are based on a number of premises, which greatly limits their application in real environment. These are 1) Training data is required to be explicit feedback. But in practice, the system may not be designed to collect explicitly ratings but a few user behavior data such as clicks, long browsing.2) with the era of big data’s coming, system size has been increased exponentially. Single processor workstation, in terms of computing power or storage capacity of view, is increasingly unable to meet the requirements.This paper proposes and implements a distributed memory based matrix factorization algorithm, while the characteristics of implicit feedback data model has been modified to overcome the current matrix factorization algorithm faced. The main work is as follows:1. Modified the traditional model which base on explicit ratings to make it work with implicit feedback data.2. Analyzed the potential parallel optimization space in alternating least squares method.3. Proposed three schemas of distribution and caching strategies, and use the api provided by spark to implement the distributed-memory based parallel alternating least squares method.We also design experiments to verify the results. Experimental results show that the new model proposed in this paper can effectively take implicit feedback data for training, while taking advantage of the distributed memory computing framework. Compared to traditional Map Reduce programming paradigm, the implementation has achieved significant performance advantages.