Research For Personalized Recommendation System

It's difficult for end-users to select useful information as data increases on network. So, a search engine with recommendations will be a gospel: Users only need to input simple query words to fetch the information they need without remembering any web URLs. Different users get different search results even if they input the same query words. One of the factors to improve a personalized search engine is “recommendation algorithm”.The research subject of the thesis is personalized search engine based on big data platform. Firstly, model ‘internet corpus' with LDA model, transfer the similarity computation of query words and text to comparison between the corresponding subject headings. It reduces the comparison items. Above all, it rises to semantics from comparison between words. It is closer to human thoughts. Secondly, we analyze the users' query logs and raise the following questions: Which factors decide the users' satisfaction of clicking the links? What will benefit from formal users' query records? How to divide user query logs? The thesis provides the following solutions: 1) The satisfaction of clicking the links is decided by the following parameters: ‘rank of the URL in the result set'; 2) There is no need to recommend if the query work is top word. Otherwise, recommend the links which the user satisfies from the user's formal search requests; 3) Partition the logs with ‘Session + sliding window'. Finally, setup recommendation system which is suitable to old and new users: use online recommendation with ALS model for old-users. For new-users, no any records in query logs, in other words, we cannot use the existent models, so, the way of old-users does not work for new-users. We should start from query words and provide objective recommendation. In the thesis, user queries can be divided into the following categories: navigation, hotspot, user feedback, and minority. Only the third category needs recommendation: provide the suitable recommendation list with the user satisfaction of clicking links. System is setup on big data platform based on Hadoop and Spark.Experiments were verified for completeness and accuracy and got better performance than benchmarks. Of course, this is not a perfect application. At the end of this thesis, we got an objective conclusion: deficiencies, thinking and vision of the future.