Performance Optimization Of JTang Distributed File Storage System

JTang distributed file storage system is a part of the major national scientific and technological projects, targeted for cloud computing and Internet-oriented application. With the upgrade of the system, we studied the performance optimization of distributed file storage system, including load balancing, replication mechanism, system expansion and distributed metadata management.Firstly, we analyzed the existing distributed system architecture, and then parsed the performance issues of JTang distributed file storage system version 1.0, at last proposed a new architecture-JTang distributed file storage system version 2.0.Second, we provided an adaptive dynamic load balancing algorithm. It combined the use of static load balancing algorithms and dynamic load balancing algorithms. It adopt to a simple algorithm in Load Balancing Module, but Data Server used its self-monitoring ability to achieve adaptive feedback mechanism. This solution not only reduced network overhead, but also lowered the cost of real-time. The experiment showed it worked well.Third, we designed a low-overhead optimistic replication. It provided state transfer mechanism and push-based propagating operations. It split file meta-data into two parts:intrinsic properties and control properties so that the data servers need not to query the master server for replication. This can not only reduce network overhead, ease the pressure of the main server, but also make the data server more intelligent.In addition, we also provided a scalable architecture-the cluster-level expansion, removing data bottlenecks by splitting the data horizontally. The directory-based consistent hash can achieve high access efficiency, data distribution (load balancing), minimize the side effect of data migration and multi-tenant isolation.