| The effectiveness of the software testing process is determined by several artifacts used in testing, including the program, the set of tests, and the test oracle. However, when evaluating software testing techniques, the influence of some testing artifacts is often overlooked, with much of testing research focusing solely on the test inputs selected. This oversight represents both a potential problem, and a potential opportunity. If there exists interrelationships between these artifacts, a failure to consider these relationships represents a threat to the validity of software testing studies. However, by understanding these potential relationships, we may discover insights that allow us to improve the testing process.;In this dissertation, we explore the interrelationship between three testing artifacts: the program, the test oracle, and the test inputs, establishing a foundation for understanding how they interact. We provide two core contributions towards this goal. First, we propose a theoretical framework for discussing testing based on previous work in the theory of testing. Second, we perform a rigorous empirical study controlling for program structure, test coverage criteria, and oracle selection in the domain of safety critical avionics software.;We find that these interactions are both theoretically and practically present, with contributions including: an improved conceptual framework for discussing software testing; a demonstration of errors in existing theoretical work on testing related to test oracle assumptions; demonstration that program structure strongly impacts both the cost and effectiveness of structural coverage criteria, with negative implications for current testing practice; results indicating that considering the interactions and tradeoffs between test oracles and test inputs may yield improvements in testing practice; demonstration of how program structure influences test oracle effectiveness, thus indicating methods of improving observability may improve testing effectiveness. These results thus highlight the importance of understanding the interaction between testing artifacts, quantify how these interactions occur in avionics software, and point towards future directions in testing research. |
