Crowdsourced Test Report Mining And Evaluation

Crowdsourced testing is an emerging technique for software testing,and it has attracted a lot of attention,from both industry and research.Crowdsourced testing is the process of an organization crowdsourcing their tasks to a potentially undefined,geographically dispersed large group of online individuals(crowd workers).In crowdsouced testing,workers help developers perform testing and submit test reports,developers must manually inspect and evaluate the submitted test reports.Due to the larger quantity and the widely varied quality,developers encounter a series of difficult problems which seriously influence their productivity and efficiency.Therefore,we attempt to conduct an in-depth analysis and evaluation on crowdouced test reports to help developers handle test reports more efficiently.This thesis mainly conducts research from two aspects.On the one hand,we focus on decreasing the manual inspection cost for developers by reducing the number of inspected test reports.On the other hand,we attempt to analyze the quality of test reports to help developers improve the inspection efficiency.In this thesis,we conduct the following work:(1)To help developers reduce the inspection cost of crowdsourced test reports,we issue a new problem of fuzzy clustering crowdsouced test reports.Aiming to resolve this problem,we propose a new framework named test report fuzzy clustering framework(TERFUR).First,two heuristic rules are designed to filter out invalid test reports.Then,TERFUR adopts natural language processing(NLP)techniques to preprocess crowdsourced test reports.Finally,a two-phase merging algorithm is proposed to implement the fuzzy clustering for crowdsourced test reports.Experimental results over five datasets show that TERFUR can cluster test reports by up to 78.15%in terms of Average P,78.41%in terms of Avcrage R,and 75.82%in terms of AverageF1.Meanwhile,the results also show that TERFUR can identify 95.33%of invalid test reports.(2)To help developers determine the inspection sequence of crowdsourced test reports,we present our attempts towards resolving the crowdsourced test report prioritization problem.We propose a new test report prioritization technique based on classification.First,we apply NLP techniques to preprocess test reports.Then,a diversity strategy and a classification strategy(DivClass)are combined to prioritize test reports.We conduct experiments over the five crowdsourced test report datasets to validate the effectiveness of DivClass.Experimental results show that DivClass can achieve 0.8921 in terms of APFD(Average Percentage of Faults Detected)and outperform comparative algorithms by 16.42%.Meanwhile,compared against existing methods,the results also indicate that DivClass can reduce the number of inspected test reports by 63.74%.(3)To help developers predict whether a test report should be selected for inspection within limited resources,we issue a new problem of crowdsourced test report quality assessment.To efficiently resolve this problem,a crowdsourced test report quality assessment framework(TERQAF)is proposed.First,we use some desirable properties to character:ize a test report.Then,a series of quantifiable indicators are defined to measure the desirable properties.Last,we transform the numerical values of all indicators into nominal values(namely good,bad)by step transformation functions and aggregate the nominal values of all indicators to predict the quality of test reports.Experimental results over the five test report datasets show that TERQAF can achieve 88.06%in terms of accuracy and outperform comparative algorithms by 23.06%in predicting the quality of test reports.(4)To help developers improve the quality of inspected test reports,we issue a new problem of test report augmentation by leveraging the additionally useful information contained in duplicate test reports.We propose a new framework named Test Report Augmentation Framework(TRAF)towards resolving the problem.First,NLP techniques are adopted to preprocess test reports.Then,three strategies are proposed to augment the different field information of test reports.Finally,we visualize the augmented test reports to help developers distinguish the added information from the original information better.Experimental results over the five datasets show that TRAF can achieve 98.65%in terms of NDCG with respect to ranking sequences and identify the valuable sentences by up to 83.58%in terms of precision,77.76%in terms of recall,and 78.72%in terms of F-measure.