Large-scale collection of application usage data and user feedback to inform interactive software development

The two most commonly used techniques for evaluating the fit between application design and use—namely, usability testing and beta testing with user feedback—suffer from a number of limitations that restrict evaluation scale (in the case of usability tests) and data quality (in the case of beta tests). They also fail to provide developers with an adequate basis for: (1) assessing the impact of suspected problems and proposed solutions on users at-large, and (2) deciding where to focus scarce development and evaluation resources to maximize the benefit for users at-large.;This dissertation demonstrates technical and methodological solutions to enable usage- and usability-related information of much higher quality than currently available from beta tests to be collected on a much larger scale than currently possible in usability tests. Such data is complementary in that it can be used to address the impact assessment and effort allocation problems in addition to evaluating and improving the fit between application design and use.;This research has been subjected to a number of evaluative activities including: (1) the development of two independent research prototypes at the University of Colorado and the University of California, (2) the incorporation of one prototype by independent third party developers as part of an integrated demonstration scenario performed by Lockheed-Martin Corporation, and (3) observation and participation in two industrial development projects, conducted at NYNEX and Microsoft Corporations, in which developers sought to improve the application development process based on usage data and user feedback.;The approach described herein involves a development platform for creating software agents that are deployed over the Internet to observe application use and report usage data and user feedback to developers to help improve the fit between design and use. The data can be used to illuminate how applications are used, to uncover mismatches in actual versus expected use, and to increase user involvement in the evolution of interactive systems. This research is aimed at helping developers make more informed design, impact assessment, and effort allocation decisions, ultimately leading to more cost-effective development of software that is better suited to user needs.