Design And Implementation Of Cloud Storage Engine

In recent years,with the rapid development of Internet technology and widely used,human activities,every moment there will be a lot of data generated and into the computer system.We store,analyze and mine these data and provide them based on these data A variety of diversified,personalized applications and services,which is already the era of cloud computing.Cloud storage as a cloud computing extension and development,its essence is to provide users with network-based online storage services,in the context of large data,the storage of massive data is the most basic requirements of cloud computing,most of the current use of distributed Environment for organizational storage.However,as the amount of data exploded,and the increase in data requirements,the current distributed architecture is also faced with high scalability and high availability challenges.This paper mainly studies and implements a distributed file system,and based on the system provides a cloud storage engine.The system based on HDFS and Ceph,through the metadata node(Name Node)distributed research,and finally put forward a good scalability of the structure,making the system theory with support for hundreds of millions of file data storage ability.At the same time,through the research and analysis of some systems with good usability,a high availability architecture suitable for the system is proposed.The system improves the network bandwidth utilization by using the Libevent + thread pool to improve the high concurrent asynchronous I/O processing of the network response and use the pipeline to write data technology.The files in the system are distributed in the form of blocks in the data server node(Data Node),each block through the redundant storage in other data nodes to save a copy,thus ensuring the security of the file,to prevent the node downtime When the file is lost.Finally,the system also supports the registration of new nodes and dynamic growth of the service,with good scalability.The main results of the work are:1.design and implementation of a distributed file system,and provides acomplete set of cloud storage engine solutions.2.to achieve the cloud storage engine file-related operation module function.3.to HDFS and Ceph file system based on the realization of the distributed system metadata server scalability transformation,to achieve the purpose of mass data storage.4.in the cloud storage engine using a high availability of the architecture design,to ensure that the distributed file system single point availability.5.the use of redundant storage mechanism,cloud storage engine to achieve a copy of the consistency of management goals.