Sorrento: Toward a self-organizing storage cluster

Recent architectural advances in terms of CPU, network and storage have enabled a new class of data-intensive applications and services. Conventional file and storage systems are unable to meet their requirements of storage virtualization, incremental expandability, high availability and self-tuning. This dissertation investigates techniques in building a storage cluster called Sorrento. Sorrento differs from previous work by focusing on the management aspect of the system and its design follows the principle of self-organization. Specifically, Sorrento virtualizes distributed storage resources as expandable volumes, which employ an object-based device model. Sorrento automatically detects node joins and departures through multicast-based membership management. Sorrento's data location scheme is able to adapt to the changes of the cluster environment and allows data to be placed on any node without restriction. Data can be quickly located using only locally maintained states. Sorrento relies on replication to achieve high availability, and replica consistency and concurrency control are managed through versioning. Sorrento automatically adjusts the placement of data to balance both storage usage and I/O workload among cluster nodes, or to exploit access locality for applications with partitioned dataset. A prototype system has been implemented and the evaluation using microbenchmarks and application trace replay demonstrates the promise of self-organizing storage clusters for data-intensive applications.