Design And Implementation Of Rapid Development Framework For Focused Crawler

At present,The explosion of the Internet,the rapid development of e-commerce,has brought the huge amounts of data.In the face of this massive,heterogeneous data,various e-commerce sites need to get data from their own platform and other platforms for relevant information.In order to solve this problem,you need to use the subject-oriented focused crawler technology.At the same time,with the rapid development of the market,followed by the doubled demand of crawler development engineer,the increasing of the crawler development engineer's salary.In order to solve this problem,this paper proposes and implements the rapid development framework of the focused crawler,which can reduce development engineer learning cost,development cost,and improve development quality and speed of development.The main technology includes the c#,the IOC,the MVC,HTML,JS,XML technology,etc.Firstly analyses the rapid development framework,development background and significance of the crawlering the network of jd.com and the network of taobao.com;Secondly,do the overall design including system platform selection,software structure design,system function module partition and data structure design;Again,the impletion of rapid development framework and the crawlering of the network of jd.com and the network of taobao.com and other networks;Finally,according to the need and the design of functional,the technical indicators of development framework is verified,and made the statistics of the development framework study costs and development costs to verify its rapid development.The Rapid development framework designed in this paper in line with the design principles of high cohesion and low coupling,which has the very high value of reuse,can significantly reduce the development cost of the e-commerce sites crawlers application,and can improve the efficiency of development accordingly.