A case study of evaluation practice in the design process of a performance support system

Electronic Performance Support System (EPSS) literature lacks a clear role for and definition of evaluation in the design and development process. Adelman and Riedel (1997) point out that the process of creating knowledge-based system is often imbalanced. They argue in most knowledge-based systems, development is often technology-driven instead of requirement-driven. It was suggested that designers must recognize that evaluation is a critical link in the application of requirement-driven development models because it provides the information that keeps the iterative developmental process on track.; This study decided to take a closer look at the complex developmental process of an EPSS in a non-profit organization. The purpose was to describe roles and functions of evaluation during the process of building an EPSS. The primary research question was "how did an ongoing evaluation facilitate the development process of a performance support system?" This study adopted the case study research as its inquiry method. The researcher served as a participant observer. Data was collected through interview, observation, field notes and document collection; data analysis was ongoing during the duration of the data collection and the final writing.; Major findings were: (1) the sequence of evaluations is used to locate the most likely source of system errors, (2) informality of the evaluation protocols does not seem to influence the quality of decision-making, (3) complexity of the evaluation depends on the relevant expertise of the designers with similar designs, (4) the pattern of decision-making in EPSS design involves a blend of confirming matured designs gained from prior experiences and exploring immature designs created for the new system, (5) tacit organizational understanding of professional practice influences the evaluation design, (6) complex organizational dynamics provide contributing forces and inhibiting forces influencing the speed of the process, (7) evaluation practice is a patchwork history of the designers' knowledge and experience, (8) collaborative decision-making in a complex system design, such as an EPSS, can be inelegant and uneven, and (9) communication is as important a function of evaluation as is decision-making.