Research On Storage Optimizationsand Applied Technologies Based On Flash Characteristics

Storage media is the foundation of computer storage architecture. From tape, floppy disk to mechanical hard drives, every progressive of storage industry had brought further development in architecture and system level. In recent years, with the declining cost of flash memory chips, a variety of storage devices based on flash memory are widely used.SSDs and other forms of flash devices are widely used to replace traditional mechanical hard disk in I/O intensive application environments because they are compatible with traditional storage systems, and with advantages like high performance, low power consumption and etc. The internal flash translation layer(FTL) in such equipment make its internal flash memory features to be transparent to the upper layer systems, while traditional file systems and database systems regard these flash devices as normal block devices, which resulted in redundant design, policy conflicts, lack of coordination and other problems. To solve these problems, this paper studied storage optimization based flash characteristics from four different scenarios including transaction processing, parallel I/O, storage intrusion detection and cloud storage cache. The main achievements and contributions of this paperare summarized as follows:(1) For transaction processing scenarios, we proposed a software-hardware collaborative transaction processing mechanism using flash characteristics named Mobius. The mechanism utilizes the out-of-place update feature of flash memory and supports traditional ACID database transactions and light transactions in log file systems within the device. And it proposed the concept of static and dynamic transactions. Mobius extends the traditional device interface, and directly provides corresponding transaction processing interface for different types of transactions from the upper layer systems. For static transaction recovery, we proposed a commit protocol based on directed acyclic graph topology relationships between static transactions. The commit protocol is a type of lazy commit protocol, which can avoid additional writings for commit record. In addition, during the recovery procedure, Mobius can skip most transactional data scanning, thereby reduce off-line recovery time andimprove data availability.(2) In parallel I/O environment, a single SSD with write cliff can cause whole system performance degradation, to against this problem, we proposed a fine-grained adaptive data placement strategy. SSD write performance will be a sharp decline under scenarios like write-intensive workloads or with less available space and etc. This problem is called write cliff. In parallel I/O environments, a single SSD with write cliff will affect the whole system I/O performance. The adaptive fine-grained data placement algorithm we proposed in this paper will carry out a fine-grained data migration between I/O nodes after write cliff happens, which can avoid the overall I/O performance degradation.(3) For storage intrusion detection scenarios, we proposed a flash-based storage intrusion detection mechanism named IDSSD. The mechanism uses flash memory to improve the detection efficiency by utilizing FTL to query intrusion detection rules, and also use the out-of-place update feature provides asynchronous flash intrusion detection intrusion detection in garbage collection procedure, which can improve the detection performance. In addition, by using out-of-place update feature of flash, IDSSD can also prevent data overlap that may occur after the intrusion detection occurs, thus avoiding the problem of lost traces of intrusion. Compared with real-time file-level semantics reconstruction method, IDSSD has improved greatly in performance.(4) For the cloud storage environment, we designed and implemented of cloud-based storage system cache using USB flash drive. This application technology uses cloud storage to extend capacity of USB flash drive while using USB flash drive as the cloud data cache, which enables cloud storage for portable data storage usage. The system utilizes the I/O capabilities of USB flash drive as a local cache while using flash drives with hardware ID and user password as the key to obtain cloud data which can ensure data security.In addition, the internal mechanism of the system using DO-LRU cache replacement algorithm to effectively reduce the cost of cloud storage.