The Implemention And Design Of Web Crawler For Price Comparing Shopping Platform

With the popularization and development of information technology. Internet has gone into everywhere of people’s lives and work, and the search engine has been the most convenient tool for people to obtain information, online shopping has become a way of life. Porducts for online selling have thousands of varieties, big different price and services Consumers have to spend a lot of time in browsing merchandise and compare price, therefore many users want to have such a system to help them complete the purchasing of goods, the system should include the price and information of most og products selling in online shopping websites. You are able to know which sites selling goods cheapest and has best services through this system. Comparison shopping platform is a good solution. But how to obtain such huge product data and price information is the most importment issue. Based on the above background, the paper proposes a solution to their data sources-Design and Implementation of Web crawler.This paper focuses on how to design and implement the web crawler, how to expand some fuctions and develop new features by Heritrix web crawler. I will do some research from the below points:(1)To determine seed links: the entrance of web crawler crawler(2)Web crawling: save web pages to a local folder(3)To analyse and extract web content: extract product attributes into a text file(4)To structure and store data: extract and store product attributes into the database one by one(5)To display products detailed information and show the comparsion result.