The Research Of The Personalized Recommendation System Optimizing & Big Data Processing

In the Information Retrieval time, people can get access to the information by searching in the internet, but whether the information is usefulness or not should be determined by themselves. In the Big Data time, Personalized Recommendation system refers to that computer carries out targeted information delivery by analyzing the user’s personal preferences and tracing his history on the internet. At present, there are many mature applications with the Recommend System in the e-commerce, music and movie industry.Nevertheless, the recommendation system still has more roundly and deeply research on system architecture and algorithm aspect.Cold start is one of the innate core problem in the recommendation system. For Recommendation System, because it can’t learn about the personalized feature of new added users or items, so it fails to provide appropriately recommendations result. The traditional method to solve cold start recommendation based on demographic has problems with overall scalability and computational efficiency, and there is the long tail effect at the same time. The paper mainly studies how to solve these problems,and it has designed a optimization scheme based on clustering and the way of information retrieval.Big data processing is the basic framework to deal with massive data. So, the paper discusses and gives a general scheme for content recommendation and collaborative recommendation as well as a big data processing scheme for the cold start.SlopeOne is a very simple and efficient recommendation algorithm in the collaborative filtering algorithm. It relies on a large number of user ratings, and it also needs a lot preratings for items to predict which means that the algorithm has cold start problem too. Aim at the first issue, the paper designs a scheme which has implemented and optimized the SlopeOne algorithm based on hadoop which is a big data processing platform. Considering SlopeOne’s increment calculation feature, the scheme come out some implementations for the feature, for the memory crash error, we also give some method to solve it. Regarding the second problem,we make full use of the first part of this paper and divide the SlopeOne data set into two categories: the train data set and test data set, The train data set is from existing historical records, and the test data set is generated from the appropriate score data set in the cold start schema. Firstly, we want to solve the SlopeOne sparsity problem, as SlopeOne cannot calculate the predicted score if the projects to predict have no task evaluation information.Secondly,in the process of big data processing,the main target is that how to make an appropriate model from the history records. As users would have requirement for the items to predict, the cold start scheme can meet the requirements perfectly, and it also reduce a considerable amount of data for calculate prediction.