Research On Several Key Issues Of Crawler In Search Engine

For the special properities of crawler in network, this paper proposes different frameworks in order to deal with the special properities of crawler. The main innovation contributions of this paper are listed below:1. This paper summarizes the basic properities of crawler in search engine.This part introduces the first basic properity of crawler in search engine- multithreading breakpoint resuming. Then the paper illustrates this properity of crawler through three aspects: background, realization, advantages of design. Furthermore, this part also introduces the second basic properity of crawler in search engine- intranet customized. And then the paper illustrates this properity of crawler through the above three aspects either.2. This paper summarizes the special properities of crawler in search engine. This part introduces six special properities of crawler in search engine: data classified in bulletin board, paralleled data-related in bulletin board, automatic dynamic updating in forum, same topics' location-tracking in forum, video-previewing in video sites, finding video's true url in video sites. What is more, the paper illustrates the six special properities of crawler through three aspects: background, realization, advantages of design.3. This paper provides the realization of eight different crawler systems. This part provides the frameworks and analysis of crawler systems dealing with the two basic properities and six special properities.4. This paper supplies the experiments and conclusions of eight different crawler systems.This part supplies the experiments of the eight different crawler systems and makes a conclusion.