Research And Implementation Of Parallel I/O And Out-Of-Core Storage Strategy On Linux Clusters

As the rapidly improved performance and gradually reduced price of workstations, following the high speed network products coming out, a new kind of parallel calculation system appears naturally. Cluster of workstations which belongs to MIMD system of distributed storage plays an important role in large scale parallel calculation .Cluster is composed of many workstations which are connected with certain kind of network. It can make use of resources of all workstations, schedule applications all over the cluster and manage the workstations in harmony. Linux is an operating system like UNIX that can be used and published freely. It is mostly used on computers based on Intel X86 CPU. Linux operating system is designed and implemented by thousands of programmers all over the world, whose objective is to create a UNIX compatible operating system which is not restricted by the copyright of merchandised software and can be used freely all over the world. Under this background, cluster of linux workstations comes forth at a very fast speed. In high performance computing realm, there are large amounts of data to be disposed. In order to satisfy the need of disposing large amounts of data, we mainly research and implement parallel I/O and out-of-core storage strategy in this paper.Parallel I/O is a very wide realm, including hardware system, the supporting of operating system, the supporting of programming language, compiler and runtime library, the analysis of I/O characteristic and performance, I/O intensive applications. In this paper, we emphasize the design and implementation of parallel file system and out-of-core storage strategy on linux cluster. First, the architecture of parallel I/O is expatiated, which is composed of disk storage system, RAID, iSCSI, file system, interconnection, network file system, parallel file system and parallel I/O interface. Second, we will introduce a parallel file system for linux clusters named PVFS(Parallel Virtual File System).Because the source code of PVFS is open and multiple application programming interfaces are supported, PVFS has a good future. Last, we bring forward a brand-new resolution for out-of-core storage problems. In the resolution, we implement a template class for one dimension arrays using C++, which uses multiple buffers in memory and associates with a parallel file. The template class for one dimension arrays supports buffer dynamic allocating, LRU arithmetic, changing the number of buffers at runtime and calculating missing rate, etc. There are two methods to construct multidimensional arrays with the template class for one dimension arrays: deriving and member constructing. Both of the two methods think in object oriented programing, and use the powerful function of C++, to design and implement the out-of-core storage library.