Students' success with World Wide Web search engines: Retrieving relevant results with respect to end-user relevance judgments

Search engines are currently the most popular method of information retrieval on the World Wide Web. However, researchers have not thoroughly examined search engines as they are used and judged by novice end-users. Calumet College of St. Joseph (CCSJ) required an investigation of search engine results to reveal how Web search engines supported the information seeking of CCSJ students. This study determined the effectiveness of information gathering through six popular search engines: Excite, Google, Lycos, MSN, Northern Light, and Yahoo!. It included an investigation of the relevance of search engine results as established by end-user relevance judgments made by novice information seekers, CCSJ students. Students in seven CCSJ English classes participated in this study. A questionnaire gathered demographic data and information about students' computer use, information retrieval experience, and experience with the World Wide Web. Students searched six search engines with search topics and queries of their choice and ranked the first 10 results according to their own relevance judgments (1 was most relevant and 10 was least relevant). The Pearson Product Moment Correlation Coefficient determined what correlation existed between the relevance rankings of the search engines and the rankings of the students. Results showed a low correlation, though a test of significance determined that this correlation is not statistically significant. Therefore, currently popular search engines are not effective in retrieving results for information-seeking CCSJ students though they may be successful some of the time. No search engine outperformed the others in this experiment, though it is also evident that no search engine consistently performed badly enough to indicate that it was the poorest performer. Furthermore, the frequency with which students used the Web, online databases/indexes, and search engines was highly correlated with search success. Two issues surfaced during the course of this study: some students' lack of computer skills, and some students' inability to construct appropriate search statements. CCSJ should take action in the areas of computer literacy and information literacy (specifically information retrieval on the World Wide Web) in order to prepare these students for the increased importance of this popular measure of information retrieval in their lives.