Course Recommendation System Research Based On Web Crawler

Taken “the online education and training system based on the cloud platform” research project as the background, this paper explore the new pattern of integration of social hot spots and network teaching, to achieve the goal of the relevant courses recommended according to the latest hot spots. Aiming at the problem of obtaining news content, the paper focuses on the analysis and research of the web crawler technology, designed and implemented a news site crawler based on Scrapy crawler framework, to achieve the purpose of fast capturing structured news data. For the persistent storage of massive news,this paper uses the Mongo DB slice technology to achieve the load balance between multiple servers, and effectively ease the storage pressure of database server.For the extraction of hot spots of social concern, use the keyword extraction method based on TF-IDF algorithm to extract the characteristics of the news, and overall considers the news weight and the TFIDF value of each keywords to achieve the hot spots and ensure its accuracy and effectiveness. For the related courses recommendation, using Solr search engine technology to search from teaching platform and make recommendation, realize the organic combination of hot spots and network education, and ultimately to arouse the enthusiasm of students and improve the teaching effect.Firstly, this paper describes the related technologies of web crawler, No SQL database,text analysis and so on. Secondly, according to the project requirements of the course recommendation system, the Scrapy crawler framework, Redis and Mongo DB are deeply studied, designed and implemented the targeted crawler for news portal website. Then,discussing the hot spots extraction method and the Solr search engine technology,completing the acquisition of the news hot spots and the recommendation of related courses.At last, the system is tested and the result is analyzed, and summarized all the work.