POSIX-Compatible Cloud Netdisk System Based On SMDFS

In order to provide high throughput access to application data,the distributed file system HDFS relaxes a few POSIX requirements to enable streaming access to file system data.However,it causes that SMDFS,a mass small file system developed based on HDFS,also can't support POSIX constraints.For this reason,SMDFS can't directly be compatible with traditional upper applications in actual production and application.Much of the existing POSIX-compatible distributed file system can support the operations such as random access originally,so it can't provide an effective reference solution for SMDFS or HDFS.Based on the existing research results,the paper analyzes the current situation of the POSIX compatibility of distributed file system,the main realization of the POSIX standard and the reason why the SMDFS doesn't support the POSIX constraints.To solve the problem,a model of file stream redirection based on local storage cache and FUSE,was proposed.Firstly,it downloads the POSIX-related target file from remote file system to local storage as a mirror file.After finishing the access to the mirror file,we synchronize it to remote file system with asynchronous thread pool to maintain data integrity of the system.Also,the overwrite-interrupt concurrency control solution,which can ensure the high concurrency performance of the system,is used to solve the problem of the same file read-write conflict during synchronizing the mirror file.The model of file stream redirection based on local storage cache provides a POSIX solution for distributed file system.Secondly,for the failure of querying metadata in client when metadata was buffered in memory,the paper designs a directory-based metadata caching mechanism in client and proposes a combination strategy of event and timer,which can restore metadata consistency in a short time.Finally,based on the file stream redirection model of local storage cache,a more complete and POSIX-compatible cloud storage disk system SMDFS3.0 is implemented from small file system SMDFS2.0,which includes disk file system,POSIX-compatible system and cross-platform web system.Test shows that,SMDFS3.0 can effectively support the random writing.Meanwhile,relative to the FUSE client of HDFS,the random read performance of SMDFS3.0 improves more than 10 times,sequential read and sequential write about 3-4 times.The performance of random write can reach to 20% of the local file system.The design of the directory-based metadata cache makes the List operation efficiency of the directory improving nearly 10 times.However,due to the additional switching of kernel-mode and user-mode introduced by FUSE,the Linux Client of SMDFS3.0 has a performance penalty of about 50% compare to java interface.