Evaluation of the perceived values and benefits of the new IDHS computerized information system

The purpose of this study was to determine users' perceived value of a new system (as a surrogate measure of a successful system), by evaluating the migration of a paper-based system used by the Bureau of Youth Services and Delinquency Prevention (BYSDP) to a computerized platform. The study also tested the reliability and validity of the measures used against two theories used in social sciences---Classical Test Theory (CTT) and Item Response Theory (IRT). Total sampling of the 855 users of a new computerized information system was done. A response rate of 30% was achieved. Perceived value was measured on a scale ranging from 0 (not valued) to 100 (completely valued). Independent variables were age, gender, length of use of the system, expectations of the system, actual changes resulting from the system, content, accuracy, format, ease of use, timeliness, impact of technology on work, interpersonal hassles, personal hassles, impact of daily work flow, frequency of problems, impact on work environment, data needs met, and job satisfaction. The F value for the multiple regression model with 19 degrees of freedom was 15.30 (p < 0.0001). The R-square value was 0.744. The most statistically significant variables were ease of use (p < 0.0421), impact on daily work flow (p < 0.005), use scale (p < 0.0001) and data needs (p < 0.0254). Reliability of the tools was confirmed by CTT and IRT. However, the corrected item-total correlations values suggested that some items do not relate to the latent construct the tool is attempting to measure, viz. the personal hassles scale and the job satisfaction scale. The problem items related to: an item from the Use Scale that probed users to determine if they found ways to get around the new system; and the Job Satisfaction item related to users' satisfaction with the nature of the work they perform. The results of CTT and Rasch analyses also indicated that the tools are multidimensional. The study suggested that these tools be refined and validated further. If the items are not clear, then the results obtained from surveys will be questionable.