| Alongside rapid global advancements in Internet technology in the past two decades, information retrieval system (IRS) has swiftly grown from initially a mere research tool into a widely used application by millions of people online. Such status of IRS therefore demands high efficiency, usability, and user-friendliness. A truly suitable IRS not only has excellent algorithms, but should also be able to aid users’ information awareness and provide a comfortable user experience. However, currently the two existing IRS evaluation methods each present glaring limitations. The system-based IRS evaluation method is based on the assumption that queries and items have an independent and stablerelationship with a binary relevance. However, this assumption is contradictory to actual users’ information needs. Moreover, since traditional evaluation parameters focus on the analysis of algorithms, they neglected users’ subjective selectivity of the results, and so algorithm-focused evaluation results cannot accurately reflect actual user activities.On the other hand, interactive information retrieval system evaluation method is also suboptimal. First of all, the data utilized by such method are mostly formed from several indirect observations; researchers often misinterpret users’ subjective statements, causing a deviation in the results. Thus, such evaluation method can only explain various phenomena emerged during the retrieval process, and from it, make very conclusive evaluations without giving specific improvements. Another reason is that each interactive evaluation method is designed specifically for one IRS, and so does not allow for researchers to make reliable horizontal comparison between many information systems using only one evaluation method.After reviewing past IRS evaluation theories and models, this research discusses the features of parameters and recent research findings of both system-based evaluation methods and interactive evaluation methods separately, based on the perspective of IRS evaluation as well as characteristic parameters. In particular, this paper focuses on the research of User Engagement Scale (UES), an interactive evaluation model. Taking into consideration various factors such as method of data collection and features of user behavior, and combining special features of IRS, a confirmation of the existing UES as a suitable evaluation model was executed. Furthermore, user information retrieval behavior was analyzed, and whether system transaction log files could serve as the information source for interactive evaluation was investigated. This research utilizes both literature review and data analysis to validate the reliability of the UES interactive evaluation model in an image-searching system environment, and also integrates past research to analyze the suitability of UES application in different types of IRS environments. User behavior within the system transaction log files, which were automatically recorded during the experiments, was extracted for further analysis, including the analysis of users’ underlying information needs and the categorization of user behaviors during information retrieval. Finally, based on the two aforementioned conclusions, the correlation between system variables and UES subscales was identified; moreover, possible improvements upon the interactive data collection process and evaluation parameters were suggested. The main conclusions of this research are as follows:1) UES is a suitable and useful evaluation tool when people look for images.2) Transaction log files consist of data that could represent users’ searching behavior.3) Information needs that occurred during the retrieval process positively influence users’ retrieval behavior.4) In the widely accepted image-based IRS, Perceived Usability of the system and Aesthetics of the user interface are not correlated with whether or not users are actually attracted to the IRS.5) Users’ general information retrieval behavior is correlated with Focused Attention, Felt Involvement and Novelty.The innovation of this research is mainly reflected in the following aspects:integrated former research to validate and give suggestions to the existing UES model; Through empirical research based on features of user behavior, investigated suitability of IIRS evaluation methods and examined the possibility of combining interactive and system-based evaluation methods.The significance of this study is reflected in theoretically comparing the suitability of UES as an interactive IRS evaluation model in different IRS environments. This could aid further research to determine if such model could be used to compare different IRS. In addition, this study used characteristics of user behavior in information retrieval to redefine IRS evaluation methods, as well as expanded the significance of extracting system transaction log file. In practicality, investigating rationale of interaction behind user behavior in the system data would enhance the recycling of data, thus greatly decreasing the time and labor costs of the data collection process in interactive IRS evaluation. The development of IRS evaluation methods based on users’ retrieval behavior and information needs would help to more accurately design IRS, and therefore has great practical significance on IRS development and growth. |
