Research On Lightweight Data Security Storage Based On HDFS

Twenty-first Century today is the era of science and technology,with the deepening of exploration and the rapid development of Internet technology,the emergence of a number of large Internet Co and e-commerce sites,such as Baidu,Tencent,Jingdong,taobao.com,Amazon and so on,which leads to a phenomenon that generate a lot of data every moment.The traditional solution has been unable to meet the need of storing and processing data at present stage,so the Hadoop big data processing technology came into being,the technology in data processing has been widely used in major Internet companies,and the cloud storage is a typical application case.Hadoop is a processing platform of being developed for the large file,the efficiency of processing in big files is very high.However,in real life,for ordinary users,their data are more showing the word documents,pictures,text,screenshots and other small files,these small files are small in size and diversity and mass.In the big data environment,it will be an important research direction of cloud storage that which way should be taken to protect the user's privacy and to ensure that the user's data security.At the same time,we also want to consider a reality: a large number of small files are stored in HDFS(Hadoop Distributed File System)must be affect the efficiency of storing and processing in HDFS.The research work of this paper is mainly:First,the improved RSA algorithm and the hybrid algorithm.This article first carries on the improved RSA algorithm.Then,based on the improved RSA algorithm,make AES algorithm and RSA algorithm two algorithms mixed to encrypt user data.Second,Hadoop small file processing problems.Aiming at the problem of storing and accessing performance of small files in HDFS,this paper proposes an optimized small file processing scheme.Third,File upload.In this paper,user-local data is uploaded to the cloud storage system using SHA-1 algorithm for message digest processing,so as to ensure the consistency and integrity of the uploaded data.In this thesis,simulation experiments and analysis are carried out to validate the scheme proposed in this paper,which uses a hybrid encryption algorithm to encrypt data stored in the cloud environment.This algorithm can provide the security protection for the cloud storage users in a certain extent.