Design And Implementation Of E-commerce Recommender System Based On Hadoop

With the rapid development of science and technology,a sharp increase in the number of the global Internet users,the Internet has spread to almost every aspect of daily life,followed by the explosive growth of information content,particularly in e-commerce."Information overload" problem caused by vast amounts of data does not bring users more options,but leave them at sea and unable to choose the goods they need from the huge numbers of commodities.How to quickly find the point of interest of users in the mass merchandise and recommend them to the users has been a problem now which the online retailers pay much close attention on.In order to solve this problem,many e-commerce websites use a program adding search engine and information classification means in the site,but it requires users interacting with the website initiatively and for users whose demand is unclear,e-commerce website can not provide relevant product pages to them either.To overcome these new issues,giving users a better shopping experience,many scholars and companies have made a lot of research and proposed the concept of the recommendation system.It can be more intelligent and initiative to find the point of interest of users and recommend the corresponding commodities to the users.Users may get referral service while simply browse relevant pages without entering any key information.This can bring much convenience to customers and improve the loyalty of them to the e-commerce websites.At first,this paper describes the research status at home and abroad and the development course of recommendation system,then it introduces the core of the recommendation system: the recommendation algorithm.But the traditional recommendation algorithms have been unable to handle massive data sets and faced with the problems of "cold start" and sparse data.For these,the paper studied deeply the theory and benefits of Hadoop distributed platforms,modifying the code of the item-based collaborative filtering recommendation algorithm,user-based collaborative filtering recommendation algorithm,Slope-One recommendation algorithm and ALS-WR recommendation algorithm.So that it can do distributed computing in the Hadoop cluster.With above theoretical support and parallel implementation of the algorithm,the experiment sets up Distributed recommendation system based on Hadoop platform,and compares and turns parameters for above four categories of algorithms on the GroupLens data set.To sum up the main works as follows:(1)Build a fully distributed Hadoop cluster,which enables it to carry out mass data storage and distributed computing and do file configuration and parameters tuning to meet the needs of the experimental environment.(2)Introduce three kinds of traditional the item-based collaborative filtering recommendation algorithm,user-based collaborative filtering recommendation algorithm and Slope-One recommendation algorithm.for the problems of sparse data of the recommendation system,the experiment introduces the model-based recommendation algorithm named ALS-WR.In order to solve the bottlenecks of hardware performance encountered by the stand-alone recommendation algorithm,the experiment modifies the above four kinds of recommendation algorithm implementation code so that it can be performed on Hadoop distributed computing platform to be adapted to the challenges of massive data-processing which the modern e-commerce websites deal with.(3)Build recommendation system based on Hadoop platform,which combines the Hadoop platform and distributed recommendation algorithms,lets HDFS and MySQL as a hybrid storage structure,and by means of Tomcat and JSP technology,and is able to show to users by the way of WEB services.Depending on the different recommendation algorithm,it evaluates the algorithm performance on Group Lens data sets and analies the results by experiment which is according to the three aspects of the accuracy,recall and precision.