Research And Application Of Search Engine Evaluation Based On Test Collections Of Dynamic Growth

Search algorithm was always evaluated on a fixed set of test collections based onthe Cranfield framework. This method is more objective impartiality and the entireprocess can be reproduced. However, this method is not suitable for the search engines inproduction environment, because the documents scale increases dynamically, a largenumber of un-judgment document will lead the result to significant deterioration. Tosolve this problem and compare the two search algorithms in the dynamic growth index,we build the necessary test collections for the special fields system, and propose ansearch engine evaluation framework based on test collections of dynamic growth. Theframework is divided into3modules: data generation, evaluation calculation and resultsdemonstration.To guarantee the effectiveness of the evaluation framework, we use the design ofusing workflow engine to drive search engines regularly evaluation, select theappropriate evaluation, also propose a prediction algorithm with the characteristics of thetopic to update test collections. And also show how to use each of the calculation resultsperiod of time in the evaluation to compare. Experiments proved that the selectedevaluation is reasonable, and using prediction algorithm to update test collections iseffective. The scope of the evaluation method has been given. And finally this evaluationmethod has been recognized by business experts of knowledge system.Finally, using JavaEE architecture, we implements the search engine evaluationsystem framework, and gives the core system implementation. Although this frameworkis not fully automated, but the workload of relevance judgment and the amount ofmanual intervention has been reduced greatly, and also an intuitive visual evaluationresults for the user has been provided. Conclusion has been given to judge the quality oftwo algorithms, which provides a reliable reference when the visual result is difficult todistinguish.