Building an extensible operating system

When designing an extensible operating system, a developer must ensure that the operating system is protected from misbehaved extensions. Two kinds of protection are needed: first, extensions should not violate the operating system's interface, and second, extensions should not be able to leave the operating system in an inconsistent state.; The major research contributions of this thesis include: (1) The design and evaluation of MiSFIT, a software fault isolation tool for the x86 architecture that ensures that extensions do not violate the operating system's interface and incurs minimal overhead. (2) The design and evaluation of VINO Lightweight Transactions, a low-overhead mechanism that allows the kernel to maintain its consistency in the face of ill-behaved extensions. (3) Experiments that show the end-to-end overhead of MiSFIT and VLT protection is low, on the order of 1-2%, and the net performance gain possible from using application-specific extensions is significant, in some cases more than 20%. (4) A cost-benefit framework for comparing extension technologies and an evaluation comparing commonly used extension technologies.