| The study of information systems has grown over the past two decades to include decision support systems (DSS), group decision support systems (GDSS), and knowledge based systems (KBS) and expert systems (ES). These systems were initially conceived as standalone information technology. With the evolution of large scale and distributed systems, however, many of these standalone applications have become embedded software elements in much larger systems, occasioning a host of attendant design, development, deployment, and evaluation hurdles.; This research focuses on the evaluation challenges associated with intelligent software systems which are embedded in larger host systems. In this research, the embedded intelligent system under study is the Shipboard Piloting Expert System (SPES), a knowledge-based system which represents expert knowledge of ship's pilots in Prince William Sound, Alaska, makes real-time collision and stranding assessments, and makes recommendations to ship's captains, mates on watch, and pilots about appropriate vessel and bridge watch team conduct. This knowledge-based system is embedded within a host group decision support system, an integrated bridge system (IBS) currently in use aboard commercial tankships.; This research proposes a comprehensive approach to evaluation of knowledge based decision aids in operational environments, and applies it to the SPES. What is notable about this research is not so much that an embedded intelligent piloting system was built, where none existed before; nor that the system was deployed for a two year period in a challenging and demanding operational environment--a tank ship's bridge; nor that the system was evaluated using multiple measures in both an operational environment and a laboratory setting. The contribution of this research lies in: (1) the development of a robust, multiple measure evaluation methodology for knowledge based systems, both embedded and standalone, (2) the empirical demonstration of the requirement for an integrative assessment of the findings produced with the comprehensive evaluation methodology, and, (3) the demonstration of positive impact of KBS and GDSS technology in a complex, large scale interdependent system, using the comprehensive evaluation methodology. |
