Research And Implementation Of Real-time Detection Methods For Code Reuse Attacks

Network servers and personal hosts are often threatened by network hackers who use maliciously crafted data packets to exploit software vulnerabilities and gain system administrator privileges.Although current researchers have conducted a lot of research on software vulnerabilities and corresponding defense methods,such attacks are still one of the biggest problems in the security field.With the widespread deployment of data execution protection and W?X,attackers are forced to reuse existing code fragments in binary files.Code reuse attacks can perform arbitrary Turing-complete calculations without injecting any malicious code.Among them,return-oriented programming has become an important means for attackers to circumvent the security mechanisms of the latest operating systems.Researchers around the world have successively proposed many methods to defend and detect ROP attacks.Representative defense techniques include control flow integrity and address space layout randomization.However,most of these methods have many limitions.Either they are easily bypassed by attackers,or the detection rate is not high,or they have high side effects,such as the need for source code information,compiler support,and so on.Therefore,in response to the huge threat of code reuse attacks,this thesis studies the characteristics of the underlying hardware and gadget chain at realtime.One is a hardware-assisted detection method with low overhead and without any side information,another is a detection method with high detection rate based deep learning which can detect all type of CRA.And then,we proposes two detection schemes for code reuse attacks.Based on this,we implemented a prototype system,and verified its feasibility of the scheme by experiments.The main work of this thesis is as follows:First,we studied The process of the ROP attack occurring at runtime and found the abnormal condition of the underlying hardware performance event when the ROP attack occurred.The proportion of return instructions increased,a large number of mis-predicted return instructions appeared,and the instruction translation lookaside buffer misses abnormally increased.Abnormal increase in data translation lookaside buffe misses.According to this situation,we proposed a hardware-assisted real-time detection method of code reuse attacks named HBDROP(Hardware-based defend ROP)based on the hardware performance counter and the last branch record.Second,we studied the characteristics of code reuse attacks that require a large number of gadget chain,combined with the current development of deep neural networks,converted the detection problem of code reuse attacks into a 2-classification problem.And we proposed a detection method named Deep CRA for code-reuse attack combined with deep learning.In this thesis,benign gadget chains and real gadget chains are converted into digital matrices,which are sent as input data to the convolutional neural network for training.In the detection stage,it aims to classify suspicious gadget chains and judge whether there is an attack based on the results.Third,due to the deep learning technology used in Deep CRA,this method requires a large number of data sets for training,which is limited by the difficulty of generating a large number of real ROP attacks.So we proposed a method that can generate a large number of ROP gadget chains.This method disassembles based on the address space layout and treats all bytes as memory addresses,thus traversing the entire address space.It find the gadget that ends with the indirect return instruction firstly,and then splice the gadgets into a chain,thereby generating the data set required by Deep CRA.Finally,based on the two methods of HBDROP and Deep CRA,we designed and implemented a prototype system for code reuse attack detection on a 32-bit Linux system.In this system,real ROP vulnerabilities were used to test and verify the effectiveness of HBDROP and Deep CRA in detecting code reuse attacks.At the same time,for HBDROP,this thesis compares it with similar methods,and it is higher than similar methods in detection rate and performance;for Deep CRA,this thesis evaluates it from multiple aspects,and the accuracy is above 98%.Finally,the limitations of the two methods are discussed.