| With the popularity and rapid development of the Internet,the cost of generating information has reduced and the ways of spreading information has increased a lot,which results in a explosive growth of information.Mass information in the Internet brings users a problem called information overload.Recommendation system can help users find personalized information quickly and effectively from the Internet,which is an important way to solve this problem.Moreover,traditional stand-alone mode will be difficult to load such a huge amount of data and the computation is hardly carried out with the rapid growing of users and data.Recommendation system in a single machine encounters a serious expansion bottleneck.Therefore,the large amount of data has put forward new requirements for the implementation of recommendation system.This paper focuses on the problem of improving the recommendation accuracy and the scalability and efficiency of the system.Firstly,user characteristics have been employed in calculating user similarity to improve the cold start and data sparsity problem existing in traditional collaborative filtering.So that those who have no history score data or very sparse score data can find similar users based on their own characteristics.And then these users can obtain preliminarily predicted scores for unrated items,which can fill the original sparse data set.Subsequently,item characteristics are employed to improve the accuracy of item similarity so that the user’s preference can be predicted more accurately.The weight of user characteristic and item characteristic in similarity calculation are both determined by extensive experiments.In order to enhance the scalability and efficiency of recommendation system,this paper introduces Hadoop platform to realize the distributed data processing and uses MapReduce programming framework to achieve parallel computation,which overcomes the bottleneck of single machine processing.The experiments of the proposed algorithm are carried out under different number of nodes using Hadoop clusters.This paper has analyzed the recommendation accuracy using proposed algorithm and compared the efficiency between distributed processing and single machine processing.Finally,this paper has finished the work on interaction between front-end and back-end,data storage and display,and recommendation engine’s equipment.Experiments prove that the proposed algorithm can obtain lower MAE and improve the accuracy of recommendation a lot.At the same time,the distributed architecture of the system based on Hadoop has obtained better scalability.Compared with single machine processing,the recommendation time of distributed processing has been decreased by nearly 30 percent with the dataset chosen in this paper.It is obvious that the greater amount of data makes the higher efficiency of recommendation using parallel computing. |
