Research On Search Engine Evaluation Based On User Behavior Analysis

Performance evaluation is an important issue for Web search engines in terms of algorithm improvement, system optimization, and maintenance. Traditional methods cannot satisfy the request of search engine evaluation due to huge amount of human efforts and an extremely time-consuming process in practice. This paper study user behehavior and mine useful information to evaluation Web search engine's performance fully and automatically. The contributions of this paper are:(1) Based on interactive process between user and search engine, we present an analysis of user behaviors about querying and clicking, and mined the relationship between user behavior and need. We also analyses different user behaviors for different user need based on several types of query needs.(2) Due to the bias and noisy in user behavior, we propose a behavior model to estimate click reliability. Experimental results show that the proposed features can be separating reliable clicks from other ones, and the model effectively identifies click quality which works well for hot queries and long-tail ones.(3) This paper presents a performance evaluation method under Cranfield framework fully and automatically, and constructs an evaluation system based on user click-through behavior. MCTR model is proposed to eliminate potential and inherent bias. The results show our method produces evaluation results similar to those gained by traditional human annotation.(4) The UA Web is proposed to describe the userful information on the Web and Monte Carlo simulation process is adopted to generate near-uniform sampling page sets. Experimental results reveal some properties of the UA Web and the index profile of four commercial search engines.(5) Considering the goal and content of Web search engine evaluation, we combine the research results about user behavior analysis and search engine evaluation, propose an improved evaluation system to work fully and automatically.