The Research And Realization Of The Key Technology In The Meta Search Engine

The information in the Internet has incredible growth with the rapid development of the computer and the network. For increasing mass network information, people not only can’t feel convenient access to the information, but always feel that they consume more and more time to acquire information they need.For this reason,the search engines are bringed out.However,the technique in the search engine system is so hard that there aren’t any search engine system can run without any flaw,such as the incomprehensive and inaccuracy of the search result. With the development of the meta search engine especially the meta search engine face to users,those problems are improved currently.Meta search engine is a kind of system that provide users for information service from multiple search engines with the unified interface. The most obvious difference between the search engine and the meta search engine is that there isn’t any resource library and crawler in the meta search engine system.In fact, the meta search engine is the agent that receive the search request and do some process,then push the processed search request to some other search engines.When other search engines response,the meta search engine then process the results and show these to the users.There are three important parts of the meta search engine,such as the process of the user input,the selection of the source search engine and the process of the result.After the description of the theories and problems about the meta search,we provide some improvements in the development from three aspects by the important parts of the meta search engine on the basis of current study and theories.First of all, as the first part of the meta search engine,the process of the user input is very important,the effect of the process is obviously.For the purpose of providing all the result that user need, we use both positive participle and negative participle in the basic process of the user input to get a complete keyword set. Not only that,we combine the long timespan category tree with the short timespan to achieve the goal that both timeliness and the good performance of the system. Secondly,as the development of the search engine industry,the amount of the search engine is increasing,so the meta search engine system won’t provide the user request to all the source search engine one time.Thus,it is necessary to create some strategies for the source search engine selection. In this premise,we unite the category tree of user with the user history search record,and use the double cache method whitch use both memery cathe and local database cache to provide user with a high speed response.Lastly,by the statistics,we know that the rate of the siminarity of the web pages is up to29%,and the rate of the total identical of the web pages is about22%. Some pages are totally identical and some have tiny changes.For the meta search engine,all the web pages of the source search engine are from the internet,there are some web pages from different source search engines are the same.On the other hand,because of the methods of the extracting are different,one webpage may have different substracts.So it is very important to find a method to reduce the reduplication of identical webpage from different source search engines.There are two strategies to solve the reduplication,the first is that reduce by the URL,the title and the substract of the return result from the source search engine,the second strategy is by analyzing of the whole web page content and identify the reduplication.We chonse the first method to process the response from the source search engine,and unite it with the user-oriented method in the process of the reduplication and the sorting of the return result,but not take into account the initial order of the return from the source search engine.