Research On The I/O Optimization Techniques Of Multi-tiered Storage Architecture

With the rapid development of high-performance computing, I/O-intensive applications put forward higher requirements on capacity, performance, scalability and reliability of the storage subsystem in high-performance computer. In the era of big data, quick analyzing and processing of massive data also need functional storage system as support. Storage subsystem is a critical component of high-performance computer. The fact that I/O performance does not match the computing performance has become an important bottleneck that impede the overall performance of the system.HDD is still the mainstream of storage devices. Its performance is far behind the performance of CPU and other computing devices. In this case, there are few things to do based on the existing storage system architecture. Multi-tiered storage architecture is inspired from the storage hierarchy in computer. The compute nodes in high-performance computer are attached with local storage device. And the original storage subsystem serve as a global storage. The global storage and all the local storage together compose the multi-tiered storage architecture. The multi-tiered storage architecture can benefit from the advantages of various types of storage resources in the system. It represents the development trend of storage system in high-performance computer. In this thesis the research is based on the multi-tiered storage architecture, specifically the I/O optimization techniques in it. The major work is as follows:1. Research on the workflow of multi-tier storage architecture. Multi-tier storage architecture can do much more to improve the I/O performance than the traditional single-tier storage architecture. This thesis analyzes the existing I/O scheduling and fault-tolerant methods in multi-tiered storage architecture, then points out their problems and what can be done.2. Design and implement an I/O scheduling method based on the busy degree of I/O node. In multi-tiered storage architecture, how to take advantage of the local storage resources and reduce I/O conflicts is key issue to improve I/O performance. However, when there are multiple jobs running concurrently in the system, due to the I/O conflicts among different jobs while accessing the local storage resources, the I/O requests of a job may need to wait a long time before getting the response and return. In such case the I/O scheduling method has an important impact on the overall performance of the system. This thesis propose an I/O scheduling method based on the busy degree of I/O node and improve the I/O performance effectively.3. Design and implement a hierarchical collaborative I/O fault-tolerant method. The existing fault-tolerant methods such as RAID and erasure code are primarily intended for single-tier storage architecture. They don’t take advantage of the collaborative fault tolerance ability of multiple-tiered storage architecture. To this end, this thesis proposes a hierarchical collaborative I/O fault-tolerant method. This method achieves real-time monitoring of the I/O process of a job by constructing an I/O proxy layer. When the exception occurs, this method uses the collaborative fault tolerance ability of multiple-tiered storage resource. This method can effectively improve the reliability and availability of a multi-tiered storage architecture. This thesis take disk full and runtime read fault as examples to demonstrate the proposed hierarchical collaborative I/O fault-tolerant method.4. Verify and exam the proposed the I/O scheduling and fault-tolerant method in a simulation environment that installs the Gluster FS. Test results showed that the proposed I/O scheduling method improves I/O performance of the system compared to existing I/O scheduling methods and verified the effectiveness of the proposed hierarchical collaborative I/O fault-tolerant method.