The Research And Implementation Of Data Fusion For Meta Search Engine

The search engine is an important way to retrieve the required contents over the massive information of the Internet.However,different kinds of search engine may use different algorithms,and have different searching scope,hence the retrieved results of different engines may be different even for the same keyword searching.In this regarding,a so-called meta search engine is designed,which facilitates to make full use of multiple retrieved results from different search engines to improve the coverage and accuracy.Data fusion is the core component of meta search engine.This article aims to design a new data fusion algorithm with the consideration of dividing the factors of data fusion into document effectiveness factors and search engine effectiveness factors.Specifically,for document effectiveness,one needs to consider the original position of the retrieving results,the well-matching of the user keywords with the title and abstract,and the repetitions of the retrieved results in the member search engines.As for search engine effectiveness,the contribution of each member search engine should be considered.After it one should combine the document effectiveness with search engine effectiveness to determine the final weight of retrieving results which is used for the further data re-sort.Given the virtues of conventional meta search engine,we adopt the two-layer caching and thread-pool techniques and construct an optimal meta search engine designing scheme.Under this scheme,the combination of database caching and memory caching can avoid repeated request of server resources,and enhance the system running efficiency as well.While the thread pool can save the overhead of the frequently thread creation.In addition,our proposed URL parameter optimization scheme can effectively reduce the page-fetch time,and the consequent waiting time of the users.This article makes an implementation on our optimized design scheme,and also tests the search engine via a chrome browser plugin and verifies the accuracy of the designed scheme.