Metadata Server Cluster Load Balancing Technology

With the development of information technology, the amount of global information is growing at a CAGR of57%. Large number of data intensive applications put forward higher requests to the existing architecture of network storage. Object-based storage system physically separates the responsibilities of metadata and file content, capturing the benefits both of fast accessing like SAN and secure data sharing across different platforms like NAS. Furthermore, object-based device is intelligent and self-managed. Object-based storage has characteristics such as strong security, data sharing across platforms, high performance and scalability, and it is the future development direction of network storage. Metadata server(MDS) is a critical component of object-based storage system, which is mainly used to manage file system name space, direct user access and map files to physical storage devices. Thus, the10efficiency of a object-based storage system largely depends on the performance of metadata servers.In this paper, we analyze the existing metadata server cluster load balancing strategies in network storage architecture. In order to resolve the workload imbalance problem caused by heterogeneous performance MDS cluster, we introduce WLBS, a dynamic load balancing strategy for heterogeneous MDS cluster. We implement the WLBS and apply it in Ceph framework. In this paper, our research and development work mainly includes the following4aspects:(1) Analyze the object-based storage and traditional network storage technologies, and then summarize two key issues involved by load balancing of MDS cluster:metadata partition and MDS popularity evaluation;(2) Propose a metadata server popularity evaluation model, which is used to obtain the workload situation of servers. On the basis of request counting, this model adds the consideration about the performance of each server. Therefore, a latency-based model is proposed to quantify the server performance. The latency-based model avoids the problem of quantifying the influence factor in the "white-box" performance model, evaluates the current metadata processing capability of metadata server precisely;(3) Propose a weight-based load balancing strategy, which focuses on the load balancing problem in heterogeneous environment. This strategy uses the metadata server popularity evaluation model described in (2) to obtain the workload situation of servers, and then computes average workload level of the MDS cluster. In order to achieve the target that let workload be proportional to the performance capability of each server, the overloaded MDS will take the initiative to migrate workload by transferring parts of subtree to under-loaded nodes in metadata server cluster.(4) Build an experimental platform, and use experiment to prove the effectiveness and feasibility of WLBS in heterogeneous MDS cluster.Extensive experiments demonstrate that WLBS can make the MDS cluster keep load balance in heterogeneous environment and improve the metadata process performance of object-based storage system.