Improving The Security Of System Software In Multiple Domain Execution Environment

With the evolution of computer technology, cloud computing and mobile internet have been changing the world in areas of national defense, healthcare, commercial entertainment, etc. In the meanwhile, the huge market as well as its margin profit have attracted the interest of the security attacker. In recent years, mobile and cloud platform suffered from many malicious software attacks, putting private data, platform security and stability to a great risk.In order to enhance security of system and application, many researchers seek for security solutions from different aspects based on multiple domain execution environment and achieve good results. The multiple domain execution environment is the environments that are split into multiple isolated execution environment(called domain)and each domain has traditional isolation mechanism such as protection rings and address space isolation. Multiple domain is well fit for improving system security for its isolation nature.However, there are two important security problems that remain unsolved in today's system software leveraging multiple execution domain:? One is that the Trusted Computing Base(TCB) of the existing solutions is hug.Many security solutions have not used the multiple execution domain to improve their security and have a huge TCB. Mobile ARM Trust Zone could be leveraged to build a dual security domain to protect critical system software by means of isolation. However, this kind of hardware is not well studied. On the one hand, some systems put a simple service running in the isolated domain to provide security service to another normal domain. Although the TCB of this kind is small, it is not flexible and general. On the other hand, some researchers try to run a complicated OS in each execution domain, which could provide flexible secure service. However, it is not secure enough for its huge TCB.? Another problem is the poor performance for domain switching. many security system software use virtualization technology to provide multiple isolation domain and deploy different security service in different domain to improve security. The multiple tenants nature of cloud platform lead the whole system to multiple protection domains and hierarchy layers both in vertical and horizontal direction. As a result,the performance decreases greatly due to the heavy overhead during cross domain interaction and thus prevent the solutions from being widely adopted.In this thesis, we try to solve these two problems in the aspect of system software in the multiple domain execution environment leveraging the architecture features. We propose several security solutions with a small TCB and design a flexible and secure cross domain call mechanism that could achieve both good security and high performance. To summarize, the academic contribution of this thesis is threefold:? To solve the lack of a trusted dual domain execution environment problem in mobile platform, we build a secure kernel called T6, which provides a trusted execution environment for mobile software using ARM Trust Zone technology. T6 runs in the secure world of ARM Trust Zone and supports legacy operating systems(Android,Linux, etc.) to run simultaneously in the normal world of ARM Trust Zone. T6 has a very small TCB while providing many modern security protection mechanisms and could ensure the integrity and trustworthy of code using secure boot and dynamic application verification. Besides, we introduce how to build trusted path on untrusted operating system and device drivers using T6. Currently T6 has been used by many research facilities and companies, proving its usefulness.? To handle the poor performance issue during cross domain switching, we design and implement a flexible and secure cross domain call mechanism. We list a series of multiple domain security usage scenarios showing that the main overhead of these security systems comes from cross domain switching. We argue that it is time for research community to rethink the traditional protection domain hierarchical structure design and then present a flexible cross domain call mechanism which could achieve both better security and higher performance. To show its effectiveness, we build a flexible and secure cross domain call system using Intel VMFUNC hardware feature and the evaluation of four different security solutions shows that the flexible cross domain call mechanism could significant boost the performance of those cross domain call systems.? To show that a flexible usage of T6 could solve many real security problems in mobile platform, we present and implement a mobile advertising fraud detection and defense framework using dual execution domain, which could effectively detect and defense all well-known mobile advertising frauds.