The Application Of Artificial Intelligence In The Information Collection By Search Engine

With the development of the network technologies, the information resources in the network have been increasing rapidly. When bringing convenience to people, it also enhances the difficulty of searching right information. In this situation, the search engine appears. People can get information they want easily with search engine. One of the most important parts of search engine is WebCrawler which can get the original information from network for the search engine.By the traditional way, the WebCrawler visits all links in the pages and abstracts the correlative information and then stores it up in the database from witch the search engine searches information. This can bring some problems. Using this method, the information which is not needed will be also got and stored up into the database, but the useful information may be overlooked. Visiting all links will make the system inefficient and overloaded.In this article we use the heuristic search to get the specific information. This can reduce the links largely and make the links visited by WebCrawler point to useful information. Certainly it can make the system more effective.In addition, we add the intelligent agent to judge which kind the page is. This part will abstract the special items of the page and generate the text vector. After calculate the text vector and main vector, the system will judge which kind the page is by the result of calculation.