Research And Implementation Of Personalized Recommendation System Based On Hadoop Big Data Frame

The information overload problem in today's world is more and more prominent, there are threemature methods to treat this problem, namely site navigation, search engine and recommendation system. The way of site navigation to solve the problem isgathers famous sites and classifies them. The search engine builds an index by massive web pages and searchs the indexto solve the problem. But when the user cannot clearly express their needs, the former two is a bit weak, while the recommended system can solve these problems. The recommendation system can analyze the user's historical behavior records, so as to actively recommend the content of the potential interest to the user. But with the rapid development of Internet, the amount of information is increasing exponentially, the traditional recommender systems will be encounter the bottleneck in the calculation of mass data.In addition, the traditional recommendation system does not consider the problem of the user's interest is easy to change.To solve the above issues, in this thesis, we refer to the previous recommendation system design. Research and implement a hybrid recommendation system based on latent semantic analysis and shard-clustering. So as to achieve the goalthat is the personalized recommendation system of books in the environment of search engine.Using Hadoopplatform solve the problem of massive data processing. This thesis firstly studies the method of search engine user behavior data acquisition. To analyze the types and characteristics of users' behavior under the search engine, and use different data collection methods and standardized methods to process these data, so as to complete the work of user behavior data collection. Secondly, In view of the uniqueness of user behavior and user's interest changeable problem in the search engine, proposed the latent semantic analysis model and shard clustering model to mining user's long-term interests and immediate interests,Latent semantic analysis model is based on content to create recommendation,so it can alleviate the problem of user and book cold start, and enhance the recommendation system coverage. And the collaborative filtering recommendation model based on shard-clustering cuts the user behavior to fragment by the attributes and content, so as to extract the user interests in different periods. So it can improve the performance of the recommendation, and make the results of recommendation more novelty.In addition, according to the problem of computing the similarity of users in the process of shard-clustering, this thesis proposes a new method of computing the similarity of the mixed type data based on the user's search term. Finally, Based on the Hadoop big data processing framework, the parallel method of user's behavior pretreatment and recommendation algorithm is studied, and design and implementation of personalized recommendation system for books under the search engine.Through use the Hadoop big data processing platform, design of the parallel recommendation algorithm, the ability of the system to deal with massive data has a great improvement. Through the synergy of latent semantic analysis model and shard clustering model, the accuracy and coverage rate of the personalized recommendation system is improved.At last, the correctness of the algorithm is proved by the test of the system and the algorithm.