A Research Of Fuzz Testing Technique Based On Reinforcement Learning And Distance Guiding

Software vulnerabilities are one of the main threats in the field of information security.Especially in recent years,vulnerabilities in some basic software and systems are becoming more frequent and are causing serious risks to the software ecosystem.Therefore,how to efficiently conduct vulnerability mining and locate potential threats early has become a major challenge for security researchers.In the field of vulnerability mining,targeted fuzz testing,as one of the major automated vulnerability mining techniques,is more efficient and accurate than traditional manual code auditing methods,and can also reduce testing costs and time,and has been widely used in the testing of various software systems,finding a large number of vulnerabilities and improving software security.In directed fuzz testing,the quality of a test case is defined as the distance between the execution path of the test case and the predefined target,the closer the distance the better the quality of the test case.However,there are two problems with current directed fuzz testing tools.Firstly,the random selection of mutation locations and operations to generate new test case during the test case generation stage results in a very low proportion of generated test cases that reach the predefined target.Second,in the seed selection stage,seeds are randomly selected from the seed queue for mutation to generate test cases without prioritising the seed queue,resulting in high quality test cases that may wait for a long time before mutation operations are performed,thus reducing the efficiency of vulnerability mining.This thesis proposes a fuzz testing technique based on reinforcement learning and distance orientation,by modelling the test case generation stage of directed fuzz testing as a reinforcement learning problem,proposing a three-factor construction method of state,reward and action,using the execution path of the test case as the state space,the combination of the difference between the distance between the newly generated test case and the seed and the target code segment and the path coverage as the reward,and the combination of the variant location by using reinforcement learning algorithms to learn the impact of mutation locations and operations on the quality of the generated test cases,and using the acquired knowledge to select mutation locations and operations,the quality of the generated test cases is improved.Meanwhile,in the seed selection stage,this thesis proposes a three-stage seed priority queue to prioritise the quality of the seeds,so that the higher quality seeds get the mutation opportunity earlier to improve the efficiency of vulnerability mining.Based on the proposed technique,we have also implemented the targeted fuzz testing tool RL-AFL,which has been tested and evaluated to increase the proportion of test cases that can reach the target code segment to the total number of test cases,and to improve the efficiency of vulnerability triggering.