Efficient metadata update techniques for storage systems

The simple read/write interface exposed by traditional disk I/O systems is inadequate for low-locality update-intensive workloads because it limits the flexibility of the disk I/O systems in scheduling disk access requests and results in inefficient use of buffer memory and disk bandwidth. We proposed a novel disk I/O subsystem architecture called Batching mOdifications with Sequential Commit (BOSC), which is optimized for workloads characterized by intensive random updates. BOSC improves the sustained disk update throughput by effectively aggregating disk update operations and sequentially committing them to disk.;We demonstrated the benefits of BOSC by adapting it to 3 different storage systems. The first one is a continuous data protection system called Mariner. Mariner is an iSCSI-based storage system that is designed to provide comprehensive data protection on commodity hardware while offering the same performance as those without any such protection. With the help of BOSC in metadata updating, the throughput of Mariner has less than 10% degradation compared to that without metadata updating.;Flash-based storage is the second storage system we leveraged BOSC. Because of the physics underlying the flash memory technology and the coarse address mapping granularity used in the on-board flash translation layer (FTL), commodity flash disks exhibit poor random write performance. We designed LFSM, a Log-structured Flash Storage Manager, to eliminate the random write performance problem of commodity flash disks by employing data logging and BOSC in metadata updating. LFSM is able to reduce the average write latency of a commodity flash disk by a factor of more than 6 under standard benchmarks.;As a third example, we applied BOSC to a scalable data de-duplication system based on the incremental backups. Each input block is de-duplicated by comparing its fingerprint, a collision-free hash value, with existing fingerprints. A range-based block group, called segment, is the basic unit to preserve data locality for incremental backups. We propose four novel techniques to improve the de-duplication throughput with minimal impact on data de-duplication ratio (DDR). BOSC is employed to eliminate the performance bottleneck due to committing segment updates to the disk.