Monte Carlo simulation-based evaluation and refinement of rule-based system

This dissertation describes new techniques for empirical evaluation and refinement of rule-based systems for classification. We develop monte-carlo simulation-based techniques for evaluating rule-based systems. These techniques enable an empirical analysis of properties of rule-based systems such as sensitivity. The techniques work by randomly modifying the original knowledge base and observing changes in performance on sample cases. Alternative techniques can perform an empirical evaluation even when no case data is available--the knowledge base is used to generate artificial case data for simulation.;Techniques have also been developed for evaluating rule-based systems by pruning methods. The techniques for pruning rule-based systems are similar to the cost-complexity pruning methods for decision trees. The degree of redundancy and superfluity in a knowledge base is measured by comparing its performance to the performance of pruned versions of itself.;While prior research has examined refinement techniques for rule-based systems with mutually exclusive hypotheses, we develop a method for automatically refining rule-based expert systems with non-mutually exclusive hypotheses. The method uses new strategies for measuring system performance. A framework has been developed that defines a space of possible evaluation strategies within which refinement systems can select a particular strategy for measuring system performance. The framework is developed in terms of underlying models of performance measurement. A particular model is selected based on various characteristics of the rule-based system.;The refinement method uses heuristics to empirically analyze the performance of the rule-based system on sample cases. The heuristics help identify candidate rules for refinement and suggest plausible refinements by selecting from the following types of refinements--deleting components from rules, adding components to rules, changing confidence factors of rules, changing ranges of numerical components, and adding new rules. New estimators are used to evaluate the plausible refinements.;The above mentioned techniques have been implemented and successfully applied to two different rule-based expert systems. Empirical results demonstrate that the methods can be effectively used for evaluation and refinement of rule-based systems.