Research And Application Of Video Recommendation Technology Based On Hadoop And Mahout

With the rapid development of information technology, especially the popularization of the mobile Internet, the number of network videos is increased sharply. And people’s demand for searching personalized videos is also increasing sharply. How to search the content which users are interested in intelligently from the vast videos information, how to discover users’ potential interests, and how to reduce the time for searching and selection are hot topics in the field of computer and network research. Then the video recommendation technology based on big data has emerged. In recent years, video recommendation technology research has made much progress with the support of big data technology, but it is still in the initial stage. And there are still some unsolved problems as follows:(1) For reasons of the poor performance in expansibility, users’ ratings matrix sparsity and cold start, the efficiency of the Collaborative filtering recommendation algorithm is low. It is difficult to be recommended effectively in big data.(2) The calculation of the similarity between users is inefficient. When the users’ scoring matrix is extremely sparse, it is difficult to calculate the similarity between users, which causes the nearest neighbors of the target user is hard to be recognized.(3) How to deal with the vast videos data offline and combine them with online recommendation is still a problem which is need to be solved to improve the ability of dealing with big data and ensure the good performance of the real-time recommendation in the system.Considering to the problems and challenges above, some methods are proposed in this paper to improve the existing recommendation algorithms. And a video recommendation prototype system based on cloud computing and big data processing technology is developed. The main work of this paper includes:(1) Improved the data sparseness of collaborative filtering algorithm by clustering users into several sets, and then searching the nearest neighbors of each user in its own cluster. It can narrow the searching scope greatly and alleviate the data sparseness.(2) A new Mahout-based collaborative filtering video recommendation algorithm(CF_PIU) is proposed to treat with the data sparsity and poor scalability of some traditional algorithms. In the CF_PIU algorithm, the similarity between users is denoted by combining the basic ideas of User-Based and Item-Based; and then the similarity is optimized by using the correlation coefficient. Experiment results show that the CF_PIU algorithm performs better than traditional video recommendation algorithms UserCF.(3) Implemented the parallel processing of the recommendation algorithm by using the MapReduce framework in the proposed algorithm.(4) Based on Hadoop and Mahout, a video recommendation prototype system which achieves the function of video scoring data extraction, similarity computing is designed. The system supports the distributed processing.