The Research Of Distributed Price Search Engine Based On DHT

In recent years, with the diversification of network resources and people’s demand for information on the exclusive domain, the study on vertical search engine becomes more and more popular. The search oriented price is one of them. However, the existing price search engines are almost always based on centralized, when a large number of users request at the same time, central server will become a "bottleneck" and prone to become single points of failure. With the continuous expansion of network scale, research on distributed vertical search becomes more and more important. In this paper, we combined P2P technology and the vertical search engine, and designed a DHT-based distributed price search engine, discussed the focused crawler’s crawling strategy, used URL rules to judgment the relevance of webpage theme, and used XPath technology to extract web information. Then we mainly discussed how to use the DHT ideology to achieve the index’s build and distributed storage, avoiding problems that the centralized index may occur effectively.Finally, aiming at the problem of unclear and confused that presented by the existing price search engines’ search results, this paper presents an idea of clustering the search results. Through the research and analysis of the existing clustering algorithms, in this paper we improved the k-means algorithm, and clustered the search results with the improved algorithm, making the similarity of the documents in the same cluster is high, and the similarity of the documents between clusters is low. Then each cluster are described by class label, users only need to browse the class label of interest to him without the need to browse all of the results returned, which saved the browsing time and lookup time greatly.