Research And Implementation Of The Secure Cloud Storage System Based On Hadoop

With the rapid growth of the amount of data to PB level, the limitations oftraditional storage mode for large data gradually exposes in the information age. Singledata storage is easy to lose the user data and result in huge losses, so more and moreenterprises and research institutions recognize that we need a new storage mechanism tochange the status of storage industry. The cloud storage technology gives an opportunityto the change of original storage mode. Foreign institutions and enterprises havelaunched their own cloud storage products and solutions in the cloud storage boom, butthey don’t pay enough attention to the safety of the products. So all kinds of safetyaccidents occur frequently and cause huge losses for the user. Under this background,the cloud storage system security problem is becoming the focus of the industry.The paper assumes that the cloud computing providers are credible and starts fromthe point view of application security. It builds a secure cloud storage system based onHadoop through research a large number of key technologies. Firstly, the paper gives usthe research status of cloud storage system and cloud storage security. Then, itresearches the key technologies for building a cloud storage system which includes theprinciple of HDFS data reading and writing, the principle of virtual disk, the principleof file system filter driver, the principle of HTTPS secure transmission andmulti-threaded transmission principle. Then, it analyses the demands of secure cloudstorage system from both functionality and performance. At the same time, it raises theoverall architecture of the system and starts the detail design from the safety client, thesecure and reliable transmission and server. It includes two encryption mechanismdesigns of virtual disk and file system filter driver, multi-threaded transmission withbreakpoint design, storage server architecture and data processing module design. Next,the text elaborates the implementation process of the various modules of the systemfrom the perspective of code to achieve and shows the results of some importantfeatures. Finally, it tests the system from the functional and performance. The systemhas been tested to achieve the predetermined functional requirements and meet the three aspects of the demands which are responsiveness, reliability, availability.The secure cloud storage system which is designed and completed on the Hadoopuses both the virtual disk and file system filter driver encryption mechanism to ensurethe security of user data in the client. It uses HTTPS secure transmission andmulti-threaded technology to ensure the safety and reliability of the transmission of userdata in the synchronous transmission phase. It provides the monitoring mechanism inthe server side and improves the maintainability of the system through the storagecluster monitoring, file synchronization process monitoring, server performancemonitoring.