Research And Improvement On Benchmarks Of Artificial Software Vulnerabilities Based On Real-world Characteristics

With the rapid development of information technology,the increasing scale and complexity of software make it increasingly difficult to guarantee the correctness and security of various types of software.For software system security,a significant threat is software vulnerabilities,which can be exploited by the attackers and cause huge threats to individuals,organizations and governments.Therefore,in order to effectively detect vulnerabilities,many researchers have proposed different works in recent years.Among the technologies,symbolic execution and fuzz testing,which have been widely applied to the industrial and academic circles as two major vulnerability detection technologies.There are many benchmarks for evaluating the performance of vulnerability detection tools.Generally speaking,the performance of a vulnerability detection tool is evaluated by the results on a set of benchmarks.These benchmark programs can be divided into real-world vulnerabilities and artificial vulnerabilities.Among them,the vulnerabilities contained in real-world programs have many deficiencies in measuring the effectiveness of vulnerability detection tools.For example,the number of vulnerabilities contained in real-world programs is relatively small and scattered.As a result,researchers have developed artificial vulnerability benchmarks to measure the performance and limitations of vulnerability detection tools.However,the current work lacks a comparison and analysis of whether artificial vulnerability benchmarks can fully represent the characteristics of real-world vulnerabilities.If the differences between artificial vulnerability benchmarks and real-world ones are not fully understood,it is not reliable to use the artificial vulnerability benchmarks to evaluate the vulnerability detection tools and even mislead the accuracy of the results.This paper mainly studies the differences between artificial vulnerability benchmarks and real-world vulnerabilities and explores whether artificial vulnerability benchmarks can measure the effectiveness of vulnerability detection tools.Based on this,we improved the existing artificial vulnerability benchmarks to make it closer to the characteristics of real-world vulnerabilities.The main contents of this paper are as follows: 1)This article compares three mainstream artificial vulnerability benchmarks(LAVA-M,DARPA CGC,Rode0 day,a total of 4427 vulnerabilities)with the real-world memory corruption vulnerabilities(a total of 80 CVE real-world vulnerabilities).It performs the first in-depth empirical study on the analysis of benchmarks of artificial vulnerabilities with manually verifies and confirms.2)To conduct more systematic comparative research,this paper proposes a general model to describe the programs and memory corruption vulnerabilities,together with how the program properties and bug requirements influence the evaluation of the vulnerability detection tools.Following this model,we design a systematic experimental procedure,and it takes 1500 man-hours to conduct experiments and analyze the results.3)This paper provides quantitative evidence on the differences between artificial vulnerabilities and real-world ones from the properties of the program itself and the conditions(control flow and data flow)that trigger the vulnerability.We also identify the factors that how they influence the evaluation of tools.4)According to the above conclusions,this paper modifies the most widely used artificial benchmark-LAVA-M from two aspects(control flow and data flow)that affect the vulnerabilities to make them closer to the characteristics of real-world vulnerabilities.Also,the experimental results show that the existing artificial vulnerability benchmarks can not be entirely and justly used as a benchmark for measuring the effectiveness of tools.5)Aiming at how to make artificial vulnerability benchmarks more realistic to the real world and can better measure the different performances of vulnerability detection tools,this paper proposes some innovative improvements.