Research On The Massive Near-line Storage System For Super Computers

With the coming of big data era, scientific research, industrial applications, network services and other areas are produceing explosively. The current supercomputer occupies pivotal position in the scientific research and national economy, taking great application field, including oil exploration and data processing, biomedical research, aerospace equipment development, satellite remote sensing data processing, Internet financial data analysis, weather forecasting and climate prediction, numerical simulation of the marine environment, civil engineering design, new materials development, basic scientific research and so on.However, along with the expansion of the data scale, super computers are still exposed some outstanding issues in handling big data applications. Most super computers apply centralized shared storage systems(such as Lustre file system, etc.), compute nodes connected to storage systems via internal high-speed interconnection network. Usually, each Lustre system has a capacity of one to four petabytes. The actual operating experience shows that when the Lustre file system space usage is greater than seventy percent, the system will become unstable sometimes, which reduces stability and availability of the system. Now, a number of typical big data applications are proposing higher and higher demands on storage system of supercomputing center, such as a 10 petabytes even more required storage capacity, organic integration with supercomputer Lustre system and so on. So, we need to research a new kind of storage structure, build the hierarchical storage system based on supercomputers, effectively solve the important technical challenges of 10 to 100 petabytes mass data storage, and provide storage and processing services for more and more big data applications.This paper proposes a dual copy and RAID-Z Massive near_line Storage System(TH-MSS) technology solution, combined with super computer Lustre storage system, TH-MSS realizes a super computer hierarchical large capacity storage system to solve the new requirements of the massive data storage.In this paper, the main research work includes the following aspects:(1) Analysis of super computer storage system structure, researched technology about using hierarchical storage to build supercomputer multi-level storage system.(2) Analysis of the TH-MSS storage system demands, proposed a technical method based on double copies and RAID-Z of massive near_line storage system, studied the relevant technologies and methods of the data migration management.(3) Designed and realized the storage strategy optimization and resource management optimization of TH-MSS, studied using MPI to realize a multi-node parallel data transmission method, analyzed the related key technologies.(4) Applied the storage servers to construct the experimental platform and verified some experiments.