Research And Implementation Of Network Robot In Job Vertical Search Engine System

The rapid development of Web technology, network information has shown atrend of the big bang. For general search engine, many fields such as web pagecapture, content indexing and so on are facing with increasing challenges. Thecontradiction of the all-inclusion of the general search engine results and the preciseneeds of the specific areas of the users is more and more prominent. Vertical searchengine which is a professional search engine can solve the conflict in some way.Network Robot is an important part of vertical search engines. It is also the datasources of the search engine. The quantity and quality of the page captured determinesthe recall and accuracy of the results of the search engine directly. Many as theconcept, the status in the whole search engine, the history and the current status areintroduced in the article. Differences of network robots in the general search engineand in the vertical search engine are analyzed specifically. The network robot used bya vertical search engine is generally called as a Focused Network Robot. To solve theproblem known as―theme island‖, contacting the actual situation of our Job SearchEngine, we proposed a customizable Network Robot, which bases on the URL-rulesand can monitor the URL list of the Hub page real-time.Some technical details in the design of the network robots are discussed andmany methods which can improve the efficiency or save the computer and networkresources are proposed in this article.Innovations in this article，Designed and implemented a three-tier structure of thefocused network robot; Designed and implemented a strategy of resource optimizationfor focused network robot; Proposed and realized the Hublist-monitoring method tosolve the island theme problem.