System Security And Architectural Support In Virtual Computing Environment

By virtualizing the underlying resources like computation,storage and network,the virtual computing environment can realize the bottom-up hierarchical abstraction,to provide a flexible,efficient and secure storage and execution environment for terminal users and application systems of the cloud computing platform.In recent years,along with the numerous security incidents happened both domestic and abroad,the security of the virtual computing environment becomes the focus that people may pay close attention to.The potential risks brought by the features of the virtual computing environment,the attack surfaces from different layers,as well as the diverse threat models considered by various service modes,they all have posed great challenges to provide secure and efficient virtual computing environment.This thesis studies the related security problems of the virtual computing environment,firstly,it considers and discusses about the runtime environment security,data security and control flow security of the virtual computing environment,then based on the rich computing resources and security enhancement technologies,it targets at the smart terminal devices,especially smartphones,to strengthen its data privacy and runtime security.Specifically,in this thesis,we focus on the following four main topics and propose solutions to resolve these problems in different levels as contributions:Firstly,to consider about the runtime security of the virtual computing environment,this paper proposes a mechanism of efficient runtime monitoring: virtual machine introspection,which provides tamper-resistant,high-fidelity ”out of the box” monitoring of virtual machines,has many prominent security applications including VM-based intrusion detection,malware analysis and memory forensic analysis.However,prior approaches are either intrusive in stopping the world to avoid race conditions between introspection tools and the guest VM,or providing no guarantee of getting a consistent state of the guest VM.Further,there is currently no effective means for timely examining the VM states in question.Based on these problems,we are the first to propose a novel approach,called TxIntro,which retrofits hardware transactional memory(HTM)for concurrent,timely and consistent introspection of guest VMs.Specifically,TxIntro leverages the strong atomicity of HTM to actively monitor updates to critical kernel data structures.Then TxIntro can mount introspection to timely detect malicious tampering.To avoid fetching inconsistent kernel states for introspection,TxIntro uses HTM to add related synchronization states into the read set of the monitoring core and thus can easily detect potential inflight concurrent kernel updates.Meanwhile,we proposes two novel optimizations to significantly reduce the working set of VMI tools,so that a workable and efficient TxIntro system can be implemented on a commodity Intel Haswell machine that provides restricted transactional memory(RTM)support.Evaluation results show that TxIntro is effective in detecting the existing rootkits,and is efficient in adding negligible performance overhead,which guarantees the security of the virtual machines' runtime environment to a great extent.Secondly,to consider about the data security of the virtual computing environment,this paper proposes a mechanism of fine-grained privacy protection: we have performed a statistical analysis on the CVE(Common Vulnerabilities and Exposures)database,and found that the critical secrets of applications(e.g.,private keys of Nginx server)can be easily leaked out in the threat models of both external attacks(e.g.,HeartBleed attack)and internal malicious OS(e.g.,rootkit).And the previous approaches are either too coarse-grained to prevent intra-domain attacks,or require excessive intervention from low-level software(e.g.,hypervisor or OS),or both.Further,few of them are applicable to large-scale software with millions of lines of code.Based on this problem,we designed and implemented a virtualization based system,namely SeCage,to protect the critical secrets of applications with fine-grained manner.SeCage is designed to work under a strong adversary model where a victim application or even the OS may be controlled by the adversaries.SeCage combines static and dynamic analysis to decompose monolithic software into several compartments,each of which may contain different secrets and their corresponding code.Following the idea of separating control and data plane,SeCage retrofits the VMFUNC mechanism and nested paging in the virtualization extension of Intel processors,to transparently provide different memory views for different compartments,while allowing low-cost and transparent invocation across domains without hypervisor intervention.We have implemented SeCage in KVM on a commodity Intel machine,and deployed it to the Nginx and OpenSSH server with the OpenSSL library as well as CryptoLoop with small efforts.It shows that SeCage can prevent the disclosure of private keys from HeartBleed attacks and memory scanning from rootkits,which guarantees the security of the application systems' data privacy.Thirdly,to consider about the execution flow security of the virtual computing environment,this paper proposes a mechanism of precise CFI enforcement: we observed that current control flow integrity(CFI)enforcement approaches cannot satisfy precision,efficiency and transparency at the same time,they either require instrumenting application executable and even shared libraries,or are unable to defend against sophisticated attacks due to relaxed security policies,or both;many of them also incur high runtime overhead.And one of the main obstacles of providing transparent and strong defense against sophisticated adversaries is the lack of sufficient runtime control flow information.To this end,we propose FlowGuard,a lightweight,transparent CFI enforcement approach by a novel reuse of Intel Processor Trace(IPT),a recent hardware feature that efficiently captures the entire runtime control flow.To address the main challenge that the IPT is prohibitively slow on the fly to decode the collected control flow traces,FlowGuard reconstructs applications' conservative control flow graphs(CFG)to be compatible with the compressed encoding format of IPT,and labeling the CFG edges with credits in the help of fuzzing-like dynamic training.At runtime,FlowGuard separates fast and slow paths such that the fast path compares the labeled CFGs with the IPT traces for fast filtering,while the slow path decodes necessary IPT traces for strong security.We have implemented and evaluated FlowGuard on a commodity Intel Skylake machine with IPT support.Evaluation results show that FlowGuard can reasonably balance the security and efficiency,it is effective in enforcing CFI for several applications to prevent existing control flow hijacking attacks,while introducing only small performance overhead.We also show that,FlowGuard can achieve a 30% optimization for the security-related metrics(AIA)of CFI,while retain the transparency of applications and deployment environment,which provides strong guarantees for the security of the application systems' execution flow.Finally,to consider about the data security and runtime security of the smart terminal devices,this paper proposes mechanisms of virtual computing environment based mobile security enhancement: the wide adoption of smartphones has stimulated a fast shift of storage,computation and interconnection from desktop to mobile devices,which also makes smartphones more dependent on the strong security.This thesis focuses on the volatility of privacy data,as well as the vulnerability of mobile runtime environment,it is based on the rich computing resources and security technologies,to enhance the security of smartphones.We designed and implemented three systems called TinMan,SplitPass and RemoteBinder respectively.For data security of smartphones,we introduced the concept of CoR(Confidential Record),and leveraged the TinMan and SplitPass systems,to protect CoR under various security threats and even physical attacks.Specifically,TinMan separates accesses of CoR from the rest of the functionalities of an app,by introducing a trusted node to store CoR,and introduces a security-oriented offloading with a low-overhead tainting scheme called asymmetric tainting to track accesses to cor to trigger offloading.While SplitPass further proposes a new password manager,which vertically separates both the storage and access of passwords into two mutually distrusting parties,so that during login,the two parties will collaborate to send their password shares to the web server,but none of these parties will ever have the complete password.Meanwhile,SplitPass seamlessly splits the SSL and TCP sessions to process on all parties,and makes the joining of two password shares transparent to the web servers.For security of mobile runtime environment,we proposed the RemoteBinder system,which supports dynamical and flexible components decouple based on the Android IPC mechanism.Driven by security requirements,it specifically decouples service and broadcast receiver components from applications,and runs the suspicious components on the third party node,which can be detected and monitored by the virtual computing environment.The above systems guarantee the data security and runtime security of the mobile devices in the strong threat models.The three proposed security solutions based on architectural extensions in this paper can be set up in the IaaS(Infrastructure-as-a-Service)and PaaS(Platform-as-a-Service)service models of the virtual computing environment,considering about the outside attacks targeting on the virtual computing environment,they are to provide efficient protection and guarantee for the virtual machines' runtime security(TxIntro),the application systems' data security(SeCage)and execution flow security(FlowGuard).The virtual computing environment based mobile security enhancement solutions can be deployed in the CaaS(Cloud-as-a-Service)service model of the virtual computing environment,considering about the attacks targeting on the mobile devices,they are to provide strongly strengthen support for the data security(TinMan and SplitPass)and runtime security(RemoteBinder)of the smartphone devices.Meanwhile,when designing those systems,besides considering about their security,this paper also makes a deep exploration on the performance,compatibility and feasibility,thus implements a series of security solutions related to the virtual computing environment with high reliability,high performance and high availability,which would have great potential applications and significance in the area of cloud computing security.