Toward high performance and highly reliable storage service

This dissertation addresses different issues related to high performance networked storage systems.; First, it introduces a new benchmark tool referred to as storage performability evaluation kernel module (SPEK) for evaluating performability of block-level storage systems. Since it runs at kernel level and eliminates skews and overheads caused by file systems, SPEK is highly accurate and efficient. It allows a storage architect to generate configurable workloads to a system under test and to inject different faults into various system components. Available performances under different workloads and failure conditions are dynamically collected and recorded over a spectrum of time.; Second, this thesis introduces a new caching structure to improve server performance by minimizing data traffic over the system bus. The idea is to form a bottom-up caching hierarchy in a networked storage server. The bottom level cache is located on an embedded controller that is a combination of a network interface card (NIC) and a storage host bus adapter (HBA). Storage data coming from or going to a network are cached at this bottom level cache and meta-data related to these data are passed to the host for processing. This new cache hierarchy is referred to as bottom-up cache structure (BUCS) in contrast to a traditional CPU-centric top-down cache. Such data caching at the controller level dramatically reduces bus traffic and leads to great performance improvement for networked storages.; Third, this thesis proposes a unified, low-overhead framework (ULF) to support continuous system profiling and optimization based on a specifically designed embedded board. ULF provides a unified interface to integrate various existing profiling tools and optimizers, and helps to easily build future tools. ULF uses an embedded processor to offload tasks of post-processing profiling data, which reduces system overhead caused by profiling tools and makes ULF especially suitable for continuous profiling on production systems. By processing the profiling data in parallel and providing feedback promptly, ULF supports on-line optimization. Our case study on I/O profiling demonstrated that ULF-enhanced profiling tool dramatically reduces overhead making continuous profiling on production systems feasible.