| This dissertation discusses the problem of software testing in (real) production environments where resources are constrained, and the testing is often driven by "business model" considerations (such as cost and time to market minimization) rather than software engineering models. In such environments the testing process is often analogous to a "sampling without replacement" of a finite (and sometimes very limited) number of pre-determined structures, functions, and environments. The principal goal is to verify required product functions to a market-acceptable level, while minimizing re-execution of previously tested functions. As a result, commercial testing is often a process that does not grow in efficiency, or grows very slowly over a number of releases.; To study this issue, this work presents a suite of software reliability engineering models specifically developed to study the above resource-constrained non-operational testing processes. The models are based on the hyper-geometric sampling function, and are formulated and validated using information from both academic experiments and real industrial project data collected as part of current work. Simulations developed for this study show that "sampling without replacement" can be a very efficient way of testing, provided test-cases are error-sensitive. Modeling and analyses confirm the "constant" testing efficiency assumption for the available case-study data. It is conjectured that, in commercial situations, the "business model" may "prompt" this effect, and that this may be a principal factor that blocks an organization from advancing on the Capability Maturity scale by discouraging within-phase feedback that might result in within-phase testing efficiency growth. However, feed-forward improvements In the testing efficiency can still take place in consecutive testing phases (e.g., from one release of the product to the next). even under such "business model" constraints. The data collected as part of current work, confirm this. The same data also indicates that the field quality of a product does appear to be positively correlated with the increases in the average testing efficiency. This may be an incentive to adopt "business practices" that allow for within-phase "learning". |
