Siddhartha: Domain-specific unit test generation for 'low-testibility' programs

This Dissertation validates the hypothesis that domain-specific language (DSL) methodology can provide essential automation support for specification-based testing (SBT) of computer program units expressed in difficult-to-test (i.e., “low-testability”), domain-specific design styles. This Dissertation presents Siddhartha, an extension to DSL methodology for development of program synthesizers to support SBT methods in novel application domains. Synthesizers map formal test data specifications (TestSpecs) into unit test driver procedures (Drivers). Both TestSpecs and Driver reference designs are represented via DSLs. Synthesizer development is iterative and example-driven. A Domain Designer applying the Siddhartha methodology in a novel application domain first selects a collection of general, example TestSpecs, then manually codes a collection of corresponding Drivers. Each Driver is expressed in a different reference design that specifically accommodates difficult-to-test, domain-specific program unit under test (UUT) design styles. After selecting the most promising Driver reference design, the set of (TestSpec, Driver) pairs become test cases for the synthesizer under development. The Domain Designer then designs a TestSpec→Driver translation function. The translation function maps TestSpecs to Driver kernels in the selected reference design. The translation function design simplifies DSL development by modularizing both TestSpec and Driver reference design DSLs into syntactic productions. This effectively optimizes DSL representations to support efficient TestSpec→Driver synthesis.; This Dissertation validates the hypothesis by answering the question “What are the relative costs and benefits of applying DSL methods to generate requirements-based and regression Drivers for a flight control system expressed in Ada?” Costs and benefits are measured with respect to manually coded Drivers, which are arguably equivalent to Drivers generated by a general-purpose tool developed by NASA. TestSpec→Driver synthesis programs developed via Siddhartha are somewhat costly, requiring between 400 and 1500 new lines of high-level programming language code. Operating such programs is relatively cheaper for both regression tests and requirement tests. Relative failure detection effectiveness for generated requirements test Drivers is the same and for generated regression test Drivers is at least as good.