Research On Buffer Overflow Vulnerability Detection Based On Fuzzing

Software vulnerabilities are now a growing problem.And malicious attackers often exploit buffer overflow(BOF)vulnerabilities to target programs.Therefore,it is important to study the detection of BOF vulnerability.The use of fuzzy testing techniques for BOF vulnerability detection is one of the most effective means of doing so,but there are currently two main problems.One problem is the poor relevance and validity of the samples generated by the fuzzy test.Another issue is the low code coverage in the sample treatment testing software that can result in BOF.This thesis addresses both of these issues.For problem 1,this dissertation first combines static analysis and deems samples that cannot trigger BOF as invalid and deletes them according to the way the BOF vulnerability is attacked,and deletes them according to an improved algorithm based on dynamic planning without reducing the basic blocks that the sample can cover.The propagation path of the test sample is then tracked in conjunction with Dynamic Taint Analysis(DTA)techniques to obtain the location of the program exception,the sample that triggered the exception,and the propagation instruction sequence of the sample.This is used to calculate the similarity of propagation paths for multiple samples,keeping only one of the samples if above the threshold and deleting the others.Finally,based on the genetic algorithm(GA)proposed in this dissertation,new samples are constructed by genetic variation operations against samples that have triggered program exceptions.For problem 2,this dissertation first analyzes the causes of the classic BOF vulnerability in detail.The target program is then disassembled,and the algorithm proposed in this dissertation is used to determine whether the function satisfies the BOF vulnerability existence condition,and to collect the vulnerable points of the BOF vulnerability in the function that satisfies the condition,resulting in a collection of vulnerable points of the BOF vulnerability in the target program.Finally,an improved algorithm based on Tabu Search(TS)is proposed in conjunction with symbolic execution techniques to deposit traversed vulnerable points in the taboo table,avoiding repeated detection of the same path while avoiding falling into a circular search.This iterates through the collection of vulnerabilities in the target application,generates path constraints for the vulnerabilities,and guides the generation of test samples to test the software.Finally,it was experimentally concluded that,in response to question 1,the method presented in this dissertation effectively reduced the number of test samples and improved the relevance of the samples.In response to question 2,the approach proposed in this dissertation ensures coverage of vulnerable points and improves the detection rate of BOF vulnerabilities.