Predicting Software Vulnerability:Text Features Vs.Software Metrics

Using vulnerability prediction model to identify vulnerable software components in a software system is a commonly used method to reduce software maintenance costs.Vulnerability prediction can help testers know the distribution of vulnerabilities in a software system,and it can help testers effectively allocate testing resources and improve testing efficiency.We can use software metrics information or text mining information to build vulnerability prediction models,which divides prediction models into tow kinds:text mining based models and software metrics based models.In order to help testers select the appropriate vulnerability prediction model,we need to evaluate the predictive power of the two kinds of models in a variety of scenarios to conduct a comprehensive evaluation and comparison.In this paper,we investigate the predictive power of these two kinds of prediction models in the context of non-effort-aware and effort-aware vulnerability prediction.To this end,we collect 20 apllications from two kinds of categories.For each application,we collect both software metrics information and text mining information,and then we use them to build vulnerability prediction models respectively.To obtain comprehensive experimental results,we conduct cross-validation and across-project prediction in both classification and ranking scenarios.The experimental results in the context of non-effort-aware and effort-aware vulnerability prediction show that:(1)in ranking scenario,text mining based models outperform software metrics based models;(2)in classification scenario,text mining based models outperform software metrics based models;and(3)most of the effect sizes(i.e.the magnitude of the differences)between these two kinds of models are trivial.These results suggest that,from the viewpoint of practical application,software metrics based models are comparable to text mining based models.Therefore,for developers,software metrics based models are practical choices for vulnerability prediction,as the cost to build and apply these models is much lower.