Distributed Memory File Caching System For High Performance Computing

With the continuous development of the design concept and manufacturing technology,the capacity and access performance of the magnetic disk storage devices have made greatprogress. Even so, the mismatch between CPU speed and disk I/O access performancebecame worse. Disk I/O is becoming the bottleneck of data-intensive computing tasks. Indistributed high performance computing environment, data is often stored in shared storagedevices. NAS storage devices, SAN storage network and RAID are adopted to improve dataaccess performance. Compared to parallel computing speed, disk I/O can always be a majorconstraint to the overall system performance. Especially when large numbers of computenodes need to access data files concurrently, the burden of shared storage system will be soheavy that causes disk and network I/O bottlenecks in the system become more obvious.Distributed memory file caching system for high performance computing managesthe free memory among compute nodes to build shared memory pool, takes advantage ofdata access locality, automatically caches file data which has been frequently accessed,and provides a unified file access interface which transparently takes access of cached dataor not, reduces computing tasks’ I/O overhead and waiting time, improves the computingefficiency and moreover alleviates the overload of shared storage system. Optimized cachefile replacement mechanism is used to improve the cache hit rate and system performance.By real-time monitoring of local host’s resource usage, future memory usage is predicted,and the size of shared memory pool is adjusted to effectively use the free memory underthe premise of not affecting the local application execution.Experiments show that distributed memory file caching system for high performancecomputing runs correctly and stably. The system effectively uses the free memory ofcompute nodes and improves the efficiency of the upper applications’ data access. The fileread throughput is promoted by a rate of200%compared to a network attached shareddisk storage system on a1Gbps Ethernet connected cluster.