Testing Machine Learning Applications With Metamorphic Testing

Recent years have witnessed remarkable progress on machine learning techniques,which have been the innovative driving force for a variety of applications.The prevalent applications of machine learning arouse natural concerns about software quality and reliability.As an important and effective mode to guarantee software quality,software testing is supposed to be applied in machine learning.However different from traditional software,machine learning software are large in scale,and their behaviors tend to evolve over time with uncetainty,making it hard to create test oracle.Also,a machine leanring software is often regarded as a black-box for its data-driven nature and the lack of methods for analyzing internal behaviors.Hence,traditional testing techniques are hard to be directly applied to machine learning software.Metamorphic testing has been successfully applied in many domains for alleviating oracle problem since its first introduction.Regarding above challenges and research gap,this paper aims to study machine learning testing methodologies based on metamorphic testing,which mainly consists of two parts:(1)A metamorphic testing approach for validating and assessing clustering sys-tems.Specifically,a set of metamorphic relations are formulated based on end users'general expectations to evaluate the ripple effect of dataset transformation.Then,a metamorphic-relation-based adequacy criterion which covers the compliance of meta-morphic relations and erroneous clustering patterns is defined.Finally,a systematic validation and assessment framework is proposed to help end users select appropriate clustering systems in specific application scenarios.Experiments on six subject cluster-ing systems and a user study have illustrated the applicability and effectiveness of our proposed approach.(2)A metamorphic testing approach for testing deep learning based image retrieval systems.Specifically,a set of metamorphic relations are defined based on the proper-ties of image retrieval systems,to validate system output behaviors.Then,in order to investigate the essential diversities of existing deep learning testing criteria,five qualitative metrics are defined to measure the internal states of nerual networks,and correlation analysis between diversities and erroneous behaviors is conducted.Finally,a diversity-guided test case generation algorithm based on greedy search is proposed,and the quality of generated test cases are evaluated in terms of classification task.Experiments on two public datasets and three deep neural networks have illustrated the effectiveness of our proposed approach.