A Dynamic Detection Technology Against "X"OP

ROP, which is short for Return-Oriented Programming, is a new attacking method based on code reuse. Different from traditional attacking methods, attackers can extract code fragments from existing libraries or executable files to form a malicious Turing-complete attacking sequence, which can modify the memory access right, create a process or steal important data. It can bypass the security mechanism provided by operating system, which brings tremendous threat to computer and network security.ROP attacking method has been developed and spread rapidly since it was proposed by Schacham in 2007. ROP attack was originally implemented on Linux 32-bit platform,and now it has been transplanted to other hardware and software platform. There are many kind of variants of ROP, such as JOP (Jump-Oriented Programming) and BIOP(Branch Instruction-Oriented Programming). This paper proposes a detection technique based on behavioral characteristics to deal with "X"OP attacks. "X"OP stands for ROP attack and its variants. The detection technique utilizes Pin (A Binary Instrumentation Tool) to detect the currently known ROP attack and its variants. The main work of this paper is as follows:Firstly, this paper discusses the principle of "X"OP attacks, and analyzes the advantages and disadvantages of current "X" OP detection techniques. This paper finds that those detection methods have a common problem, i. e., they can only detect a single type of attack. To solve this problem, this paper proposes a detection method based on the common characteristics of "X"OP attack. Meanwhile, it compares the difference of special instructions (jmp, ret, call) between ROP attack and normal program, and extracts the characteristics of "X"OP attack. According to the research, this paper presents an "X"OP detection solution based on behavioral characteristics.Finally, this paper builds a prototype system to detect "X"OP attacks, and use classic"X"OP attacks to test its functionality and performance. The experimental results show that this system can detect ROP attack and its variants effectively.