A Formalism for Structuring Evaluation Problems: The Case of the Evaluation of a National Health Informatics Initiative

This research is designed to strengthen health informatics evaluation by providing the means to systematically define evaluation problems. The evaluation problem structuring formalism was developed and demonstrated. The formalism was designed to enable defining evaluation independently of settings and study designs by using the concept of problem structuring.;A secondary analysis was conducted to assess the demonstration of the evaluation problem structuring formalism on a case from health informatics. Two evaluation reports from the case were purposively sampled using a structured document analysis protocol into sentences that yielded one hundred and sixty four statements directly addressing the objectives of the reports. Each statement was then analyzed using the formalism as a structured guide. The focus of the analysis was finding one or many object judgment pairs in each statement and determining whether or not any object judgment pair satisfies the mathematical rules of the formalism, namely that the pair is an element of the graph of a function from a set containing the object to a set containing the judgment.;The study found the demonstration of the formalism to be successful for some statements and partially successful for others. At least one statement had at least one object judgment pair satisfying the rules, indicating full success of the demonstration of the formalism for those statements. At least one statement had at least one object judgment pair but did not satisfy the rules, indicating partial success of the demonstration of the formalism for those statements. In addition to evidence of demonstration success, the study findings show that the process of using the formalism classified the evaluation problems found in the case into those involving judgments about specific objects, conditional judgments about specific objects, and overall judgments about no object specifically.;The study contributes to industrial engineering methodology by proposing a novel demonstrated formalism, detailing a feasible approach to systematically demonstrating formalisms, and showing that even failures and partial successes in formalism demonstrations are useful: they provide a classification scheme. The study contributes to strengthening health informatics evaluation by providing evaluators and clients a principled engineering approach for defining evaluation problems systematically.