The Design And Implementation Of Distributed Cloud Storage System Based On HDFS

With the rapid development of information technology, especially the mobile Internet, Internet of Things etc. Data shows the explosive growth, and we have entered the era of mass data.Traditional storage management methods no longer meet current storage status,the way to effective storage,management, and maintenance of these data has become a hot issue.The rapid development of technology of cloud storage makes cloud storage become a new type of data storage solutions.More and more developers and companies migratetheir data to the cloud platform to reduce data management, and operation and maintenance costs, and mitigate the impact of the huge amounts of data.But cloud storage is still in the development stage, a variety of technologies and corresponding laws and regulations are not mature and perfect. So the data stored in the cloud is not foolproof, likelysome unexpected events will lead to the loss of user data, or information leakage of confidential data. Taking these factors into account,the internal sensitive and important data of a company is not suitable to be stored on top of the existing commercial cloud storage systems.This articlehas made a comprehensive analysis of the current situation of the development of cloud storage technology at home and abroad, learned from the technology of the most stable, most mature cloud storage products-Amazon S3, token into account the status of the existing hardware storage devices, finally puts forward a new distributed cloud storage solution which has the characteristics of high scalability, high reliability, and compatible with different storage devices.The System contains three parts:the underlying data storage section, the intermediate logic processing section, and the front access portion.The whole system is built on top of a distributed file system HDFS, and takes full advantage of its excellent performance in data disaster recovery, fault tolerance, error correction, and data recovery. The file read-write module is designed and implemented on top of the underlying storage system. The design and implementation of the proxy module for front-end request, the core business logic processing module, and the metadata storage module based on databaseis based on the compatibility with Amazon S3agreement. The system provides two service access methods:Web front-end browser access method and SDK access method. Besides, in order to ensure the security and integrity of the data request in the transmission process, this system designs and implements the security control module. Thus, it builts a highly scalable, highly fault-tolerant, reliable, secure distributed cloud storage system. Finally, this articlecompletes the distributed deployment and testing of the entire cloud storage system.