Software Testing For Cyber-physical Systems Suffering Uncertainty

CPS are engineered systems that are built from,and depend upon,the seamless integration of computation and physical components.CPS tightly integrate computing devices,actuation and control,networking infrastructure,and sensing of the physical world.CPS technology will transform the way people interact with engineered systems-just as the Internet has transformed the way people interact with information.As an important part of cyber-physical systems,software's quality not only determines whether CPS would deliver satisfactory services,but also affects their users' life safety and properties(e.g.,Tesla crashing in its autopilot mode,self-driving Uber killing a pedestrian,etc.).Software testing is an important and widely-used approach towards software quality assurance.However,environmental uncertainty challenges the testing of CPS.Due to the limitations of human cognition of environments,environmental uncertainty exists inherently in CPS.Basically,environmental uncertainty challenges CPS testing at two major aspects as follows:?First,environmental uncertainty prevents CPS from sensing its surrounding envi-ronment accurately.Such uncertain sensing not only makes it difficult to explore a CPS's state space effectively and efficiently,but also makes it difficult to determine whether CPS are executing correctly.? Second,environmental uncertainty prevents CPS's developers from acquiring a for-mal specification of the CPS's running environment.Such uncertain specification partially leads to the wide-use of the data-driven software development paradigm,which uses machine learning techniques to learn a program's application logic based on data,instead of a given software specification.The lack of the software specification makes it difficult to test such programs for existing software testing approaches.In this dissertation,we address the challenges of CPS testing brought by en-vironmental uncertainty.We make a comprehensive analysis of the environmental uncertainty's impact on CPS testing,and propose a research framework for testing CPS effectively and efficiently.The framework consists of two parts.First,it executes a target CPS in a given environment to discover the CPS's failures.Second,it evaluates the CPS's suitability in a new running environment to determine whether one needs to re-test the CPS in the new environment.The main contributions of this dissertation are as follows:1.Proposed a test input generation approach SIT for CPS testing.The test input generation explores a program's input space to discover its failures.For CPS testing,the program-environment interactions and environmental uncertainty expand the program's input space and make the failure-exploration difficult.Our SIT approach consists of an interactive program testing framework and a white-box-sampling-based test generation technique.SIT can systematically explore a CPS program's space in a light-weight way.Experiments based on real-world CPS subjects show that our SIT approach achieves a higher failure detection rate(22.4-42.2%higher)and smaller time overhead(66.7-83.3%smaller)in testing CPS,compared with existing testing approaches.2.Proposed a test oracle generation approach CoMID for CPS testing.The test oracle generation determines whether a program's execution is passing or failing.For CPS testing,environmental uncertainty blurs the boundary between the program's passing executions and failing executions,which makes it difficult to detect CPS's abnormal states.Our CoMID approach consists of a context-based trace grouping technique and a multi-invariant detection technique.CoMID can accurately detect CPS's abnormal states.Experiments based on real-world CPS subjects show that our CoMID approach achieves a higher true-positive rate(5.7-28.2%higher)and a lower false-positive rate(6.8-37.6%lower)in detecting CPS's abnormal states,compared with existing abnormal state detecting approaches.3.Proposed an environmental suitability evaluation approach SynEva for machine learning programs.CPS use machine learning programs to develop software under uncertain specification.The "no oracle",and lack of "gold-version" problems make it difficult to test machine learning programs.Our SynEva approach consists of a program synthesis technique and a similarity measurement technique.SynEva can effectively evaluate whether the knowledge a machine learning program extracts from training scenarios still suits predicting scenarios.Such suitability reflects the overall quality of a machine learning program's execution.Experiments based on real-world data sets show SynEva approach's effectiveness in evaluating machine learning software's suitability for new scenarios.For a given machine learning program,SynEva's reported evaluation value has a strong negative correlation(-0.81)with the program's suitability for new scenarios.