Analysis Of Dangdang Information Based On Scrapy Framework Crawler And Data Mining

In the past two decades,online shopping has gone from no one to Renren,which is enough to show that e-commerce is now in a relatively mature stage.Its emergence has changed consumers' shopping method from offline to online,which facilitates consumers' shopping and brings new ways of shopping.The transaction volume on Tmall platform of the singles' day shopping festival in 2019 reached 268.4 billion yuan,up 25.71% year on year,indicating that the scale of online shopping users is expanding and online shopping is becoming more and more important in People's Daily life.For online consumers,first of all,they choose the products they intend to buy by browsing the web information of the products,which will have an important impact on the choices of online consumers and the marketing plans of e-commerce operators,if you can make good use of this information,you may be able to provide better service to buyers,and you may be able to guide sellers in their marketing decisions.Therefore,the acquisition and analysis of these web page data is of certain practical signific ance.How to effectively use the web page information of goods,mining hidden value among them.Be cause the amount of web page data is huge and its content is complicated,it is not easy to obtain the data information scattered in the web pages,so the efficient and accurate acquisition of the required web information has become the primary content of this study,and the development of web crawler technology provides technical support for the acquisition of web data in this paper.This paper will use Python to obtain the data of the e-commerce website page and analyze the crawled data with the data mining method,hoping to find the hidden valuable information,which can help the decision-making of the e-commerce operation team.The analysis of e-commerce webpage information mainly includes data crawling and data analysis.In this paper,dangdang web page information analysis of the main content:first,using python to design a framework based on Scrapy crawler,this paper introduces in detail the method of crawling the information of book title,author,sales volume and so on,and the process of st oring the crawling information in My SQL database,in order to lay a good foundation for the mining o f the following web page information.Secondly to crawl data using appropriate methods for its analysis,mainly has made a descriptive analysis of the data,data preprocessing,text vector space construction,the determination of the optimal clustering number,use the k-means clustering algorithm to analyze the text data,finally has carried on the explanation to the output,contained in the text information of value to mining.