| The cyberspace we are living in is in a severe security situation. The frequently happened cyber attacks based on information system vulnerability exploitation brings serious threats to national security and personal interests. Moreover, the rise of mo-bile internet speeds up the development of information technology, along with new challenges to traditional system and software security. However, in current security ecosystem, vulnerabilities residing information system and software are hardly elimi-nated, and existing research results on defense, mitigation and detection are fail to defeat vulnerability exploitation-based attacks due to the inefficient performance and low coverage.As the key technique of memory vulnerability exploitation-based attacks, code reuse can circumvent non-executable memory, code signing and other security mech-anisms, and perform its crafted program logic by employing code snippets present in memory. Both characteristic-based and software behavior-based detection inethod can be bypassed by its flexible organization of instructions and program logics. Therefore, we study on binary code reuse, especially the defense and detection of code reuse-based attacks, and try to carry out effective methods to minimize corresponding threats. The details of our research work in this paper are as following.Firstly, according characteristics of repeated control flow transfer and stack abnor-mity caused by code reuse in vulnerability exploitation, we proposed a stack abnormity-based method for code reuse detection and shellcode attribution. A prototype named S-Tracker is also implemented for Windows platform. In order to mount code reuse attacks, adversary should put massive malicious data into the stack or a forged stack created on heap, which may overwrite sensitive pointers unintentionally and change stack pointers out of the legal stack memory. Therefore, we figured out the definition of stack abnormity for code reuse detection and designed a sensitive system call-driven backward stack tracing method to examine the abnormity of every stack frame to dis- cover attacks. Comparing with Microsoft’s EMET, S-Tacker, implemented in Windows kernel land, is much more difficult to be bypassed.Secondly, as code reuse attacks relies heavily on crafted stack data, an ARM-specific stack layout randomization strategy for relocating stack objects is proposed. By putting more free-to-use registers into register-list in functions’ prologue and epilogue push/pop instructions, paddings with uncertain size are inserted into the stack. Consequently, adversary would fail to locate correct data objects on stack and restore them into expected registers. Thus the execution of gadget-chain would be stopped. Moreover, the feature that ARM instructions share a fixed instruction length makes this randomization strategy more practical by introducing none extra instructions or file layout changing. Though the evaluation focused on Android binaries, the idea itself is also capable for iOS applications.Thirdly, we evaluated the security of Android ART, the most recent Android run-time, from the perspective of code reuse threat. We found a flaw in ART’s ASLR implementation that fails to randomize the image base of boot.oat, and crafted a proof of concept exploitation idea. As code reuse relies on gadgets ending with branch in-structions to transfer control flow, we propose a method that tries to randomization all direct and indirect branch instructions in binaries to thwart gadgets. The prototype implemented on Android 5.1.1 proved the effectiveness our randomization idea on defeat code reuse attacks.At last, we designed and implemented a security-enhanced Android system, which systematically improved the system security and significantly minimized the damage caused by code reuse and other attacks. In current Android ecosystem, applications are suffering various attacks from untrusted shared libraries, such as backdoor, privilege escalation among application’s various code domains. By separating untrusted shared libraries from applications’ sandbox to a newly created application sandbox that is only granted with few permissions, even the sandbox for shared libraries is fully compro-mised by attackers, the damage is limited to certain permission capabilities enforced by Android sandbox, and privilege escalation via reusing privileged code is also defeated. |