The Research Of The Storage Resource Management Technology Based On Cloud

With the rapid development of cloud computing, cloud storage plays a vital role. Hadoop is an efficient data storage and analysis framework. It has become the most popular projects, whether academic or industry, in the research of cloud storage and its application is very extensive. It is the enterprise applications to provide the more stable, more extensive data distribution and higher reliability and more flexible computing power in the search engine optimization, machine learning and other aspects of advertising, more and more people and companies use.As a distributed storage and computing framework, the characteristics of open source, powerful and free of Hadoop, attracted a large number of personnel to carry out research, but it also has obvious shortcomings, such as no supplier provides support tools also no one can guarantee the quality of the code. As result of the lack of appropriate tools, the management and maintenance of Hadoop operation mainly on the terminal command line in Linux, in the course of the operation go astray will cause the cluster operation error. Every released version of Hadoop will be tested perfectly, but no one can guarantee its performance in the cluster to achieve the best, many organizations were optimized according to the cluster of their own, such as Cloudera CHP, hortonworks HDP etc..This paper is aiming at the above problems, the design and implementation of the storage resource management system based on cloud architecture. The system uses Puppet as the underlying architecture configuration synchronization, uses Hadoop as a distributed data processing and Storage Cloud Architecture, we developed the storage resource management system through the Spring MVC framework, which contains the functions of the cluster monitoring, management and optimization. This system provides a step on step deployment method, cluster nodes dynamically add and remove method and cluster replication scheduling optimization. The main contributions of this paper are as follows:1proposed a deployment framework based on the cloudAccording to the characteristics in the process of deploying Hadoop, we proposed the step on step deployment framework, which is a master-slave mode. Through the description of the system architecture and the design of the server and the client, we prove the feasibility of one click deployment of cluster on the technical level.2proposed a dynamic management model of the cluster nodesTheoretically, the cluster nodes can be increased unlimited, how to manage these nodes uniformly is one of the problems. This paper proposes a dynamic management model, which is used in the base of the observer pattern, listening on the behavior of the nodes, automatically update the status of the node when the nodes status changed, which reduce operation of the user..3proposed a cluster scheduling optimization modelHow to efficiently utilize the storage space is a hot issue on distributed storage research. Storage cluster, each data node storage capacity may not be completely consistent. When the selected data nodes close to the full, the master node will automatically load balancing, occupying the data transmission bandwidth, not only affects the performance of data transmission, but also causes the unreliable of data. This paper presents a cluster scheduling model:the first stage is calculating the ratio of storage optimization at first, then use a local optimization storage scheme based on greedy algorithm, selecting the storage nodes and balancing the space of the placed replications; the second stage is monitoring storage cluster node placement real-time, dynamic adjustment of the node to place copies, making the usage of storage resources efficiently.The storage resource management system is in use of the video cloud storage system for Ministry of science and technology, simplifying the user manual of Hadoop and abandoning the complex way of command-line management cluster, with the help of this system, the cluster can be built easily. With the simple and convenient operation, the system plays a crucial role Around the entire life cycle of Hadoop.