A Hardware Virtualization-based Isolation Mechanism For Shared Library

In the modern operating system,applications and shared libraries belong to the same memory space and there is no privilege switching between them.While this mechanism provides convenience for library function calls,it also poses a potential threat to user security.In code reuse attacks,shared libraries are the preferred target for attackers to search for malicious instructions or function interfaces.While isolating shared library into an isolated execution environment is a promising countermeasure,existing approaches either require modifying and recompiling shared libraries or breaking the shared feature.So most of them can not be effectively applied in modern systems.In order to solve these drawbacks,Libsec-an efficient and transparent shared library isolation mechanism is proposed.With the help of Intel Virtualization technology –Extended Page Table and related expansion instruction – VMfunc,Libsec firstly isolates the share library from the original application address space and creates a separate shared library address space,it can prevent the application code from jumping directly into the shared library.Then Libsec provides the corresponding address space switching interface to ensure that the correct interaction between the shared library and other components after isolation.Libsec avoids modification of the application and maintains the shared features by static instrumentation and dynamic processing combined.Finally,since the attacker does not use the special interface to switch address space,Libsec can determine the legitimacy of shared library runtime behavior through the exception events happens in the hypervisor.To demonstrate the effectiveness of Libsec,Libsec is applied in the protection of the Glibc standard library,OpenSSL library and so on.Experimental results show that the attacker can not directly execute the shared library's code,so Libsec effectively prevent the above-mentioned attacks.At the same time,the performance test results show that Libsec is used to isolate the underlying shared library with only about 15% performance overhead and appropriate memory overhead.