Cooperative Testing and Analysis via Informed Decision Making

Although tool automation is important in software testing and analysis for reducing manual efforts, tools face various problems when dealing with complex software, and such problems would be still difficult for the tools to tackle in the foreseeable future. For some of these problems, tool users may help the tools tackle these problems, such as providing mock objects to simulate the behaviors of the external libraries. With the provided mock objects, the test-generation tools can be reapplied to generate more test inputs, achieving new coverage and carrying out software testing more effectively. In this way, the users do not need to generate test inputs for all the newly explored code. If more problems are encountered in exploring the newly explored code, the users can provide their help again. To support such new types of cooperation enabled by the advances on software testing/analysis tools, there is a strong need to provide a user-tool interface that communicates sufficient and precise information to the users (e.g., reporting the problems faced by the tools), allowing the users to help the tools address the problems.;In this dissertation, our framework focuses on providing interfaces to support two types of cooperation with software testing/analysis tools for SE tasks: • Problem-Diagnosis Interface: For certain SE tasks where automated tools drive the tasks towards a goal of testing and analysis, users can help the tools address the encountered problems since the complicated characteristics of the code under test are beyond the capability of the tools. For such SE tasks, our framework provides problem-diagnosis interfaces to support a type of cooperation, called problem-diagnosis cooperation, as follows. Users first set up and apply a tool to conduct initial testing and analysis on a program. The provided interface then shows the feedback to the users, including the effectiveness of the tool and the problems faced by the tool; these problems are precisely identified and the benefits of solving these programs are accurately estimated using analyses of the program and the tool. By looking at the problems, the users provide guidance to the tool based on the feedback, helping the tool address the problems. Such process forms a feedback loop and can be repeated until the effectiveness is satisfied or the users run out of patience. • Behavior-Diagnosis Interface : For certain SE tasks where users inspect software behaviors to determine whether the behaviors are expected for achieving a goal of testing and analysis, tools reveal and explain software behaviors to help the users make informed decisions, because behavior expectation lies in the mind of the users and the tools need the users to make decisions. For such SE tasks, our framework provides behavior-diagnosis interfaces to support a type of cooperation, called behavior-diagnosis cooperation, as follows. Users first set up and apply a tool to conduct initial testing and analysis on a program. The provided interface then shows a list of behaviors related to the goal for the users to inspect, and the users make decisions on whether these behaviors are expected. Furthermore, the interface also provides explanations for the behaviors. Based on whether the behaviors are in the form of either concrete instances or abstract models, these explanations can be in the form of (1) extension: a larger scope of information to collect more concrete instances or extend the models of the behaviors, (2) instantiation: instantiations of the behaviors' abstract models, or (3) abstraction: inferred models from the concrete instances of the behaviors. Such explanations help the users make informed decisions on the behaviors to be inspected.;The key difference between problem-diagnosis cooperation and behavior-diagnosis cooperation is that problem-diagnosis cooperation typically engages the users to diagnose some intermediate results used for producing the final results, while behavior-diagnosis cooperation typically engages the users to diagnose the final results directly.;We propose a number of approaches under cooperative testing and analysis, where each approach provides the interface to support cooperation between tools and their users for a specific SE task, improving either functional or non-functional quality of software.;First, we propose approaches to better support cooperation for automated test generation, ensuring functional correctness of software. Second, we propose approaches that provide behavior-diagnosis interfaces to better support cooperation in security analysis, with the focus on mobile privacy control and security policy extraction from requirements documents. Finally, we propose an approach that provides a behavior-diagnosis interface to better support cooperation in performance analysis, with the focus on identifying workload-dependent performance bottlenecks (WDPBs).;Our empirical results show that the approaches developed under our framework provide interfaces to effectively support cooperation for the respective SE tasks. (Abstract shortened by UMI.).