Design And Implementation Of Secure Cloud Storage System Based On HDFS Small File Processing

With the continuous development of network construction,the data is showing an explosive growth,how to manage these massive data has become a hot and difficult research issue.Hadoop Distributed File System is mainly used to store large files,and the support of small file storage is not perfect.But now most of the data stored and transmitted in the network are small files,which led to the system's storage bottlenecks.In addition,security incidents in the network are also happening frequently,the reason is that the security of cloud storage system is not perfect.In view of the above problems,the HDFS storage and data attack methods are studied,and more attentions were paid to the the advantages and disadvantages of HDFS storage data and security problems.Through the intensive research of HDFS small file storage and AES,this paper design and implement a cloud storage system with data confidentiality,which support for high performance access to files and easy to expand the cloud storage system.(1)Optimizing HDFS small file processing method.This paper analyzes the solutions of small files storage at home and abroad,and summarizes the advantages and disadvantages,and puts forward the concept of small file processing module.The module is composed of three parts: file judgment,file merge and file index.It realize small file batch processing and merge.The index record is dynamically connected with the database,which avoids unnecessary connection with HDFS.Also,prefetching and caching techniques are used to further improve the file access speed.(2)Optimizing the AES algorithm.Since the encryption and decryption principle of AES algorithm is open,the encryption process and decryption process are optimized according to the principle of AES algorithm.By merging and reordering the relevant steps of encryption and decryption process,the processing speed of the algorithm is improved.(3)The HDFS system which supports the storage of small files and the security is designed and implemented,it integrates the authentication mechanism,the optimized small file processing method,the improved AES algorithm and the hybrid encryption idea.This system uses a hash function in identity authentication to ensure the security of the password.The small file processing module is used to process the uploaded file,saving the memory size of NameNode and improving the reading speed of the file.The use of improved AES algorithm to further enhance the security of the data.And the improved AES is used in conjunction with the RSA,this hybrid encryption method improves the speed of file transmission and the security of the key.