The Research And Implementation Of Cluster And Computing Resource Management In D-Ocean

In the "Big Data" era, how to exploit the massive unstructured data on the internet is a hot research topic. As the evolution of distributed computing technology, distributed architecture has become the mainstream of unstructured data management system. Resource management is one of the most important components of a distributed system. Efficient resource management could meet dynamic resource requirements of the system and then improve the service capacity and processing performance.This paper reviews the different resource management schemes in some famous distributed systems, and then designs and implements the resource management for D-Ocean, which is an unstructured data management system.Firstly, in order to improve the resource utilization of D-Ocean, a cluster management and monitoring platform is proposed and implemented. It can improve the service ability, flexibility and scalability of the system by a fine-grained role management. In addition, the management and monitoring tool in the platform can monitor the status of nodes and tasks in the cluster.Secondly, an agent-based resource management system is proposed and implemented for the distributed computing platform of D-Ocean. Computing nodes and task definitions in the platform are modeled to realize computational resources scheduling and optimization. The stability and efficiency of the system is indicated by the practical task processing experiment.Nowadays, D-Ocean can support various unstructured data application and processing tasks stably, which indicates the feasibility of our system.