Research On Real-time Monitoring Secure Data Storage System Based On HDFS And Spark

With the development of Internet and information technology,users are creating data at all times,the amount of data grows rapidly of the PB level.Data that contains a large amount of personal information should be easily access and keep security.The traditional model that stores all data sets in a single storage system,regardless of availability and reliability,is no longer applicable to current application requirements.The emergence of cloud storage technology solves this problem,reflecting the advantages of the low cost,large storage capacity,easy management and so on.But the existing cloud storage system did not solve the data security issues,a various types of security incidents occur frequently.Based on this,this paper designs and implements a secure data storage system based on HDFS and Spark real-time monitoring.Mainly includes three major functional modules,data encryption,data storage and real-time detection.The encryption module uses an existing B/S-based key management scheme,reducing the probability of data being stolen.The storage module uses HDFS that both include reliability and availability.Real-time monitoring module that uses Spark for massive data is the core module of the system,which ensures performance and security of system.After analyzed function and performance requirement of system,the requirements of the system are clarified.Three key points of real-time monitoring scheme,network attack recognition and message queue selection are studied respectively.(1)This paper presents a real-time monitoring scheme based on streaming data processing,and execute real-time analysis of each request,execute request according to the analysis result.This scheme is a true real-time monitoring method is different from the real-time monitoring method of existing program to analysis log.(2)Analyzed 12 kinds of common types of network attacks,and designed the appropriate means of identification and Spark test methods.And design a rule bank that is conveniently added and deleted for the means of identification.(3)Analyzed the characteristics of five common message queues,and then they were deployed and tested for data throughput.Then,a secure data storage system based on HDFS and Spark real-time monitoring is designed and implemented according to the requirement analysis and research results.The system is divided into front-end module,back-end module and real-time analysis module.The front-end module is responsible for interaction and encryption.The back-end module is responsible for data processing and request processing.The real-time analysis module is responsible for real-time analysis and processing of requests.Finally,a comprehensive functional and performance test of the system was carried out.The test results show that the system not only solves the problem of user login,data encryption and real-time monitoring,but also has a very good performance in real-time monitoring processing speed,request parallel processing and encryption and decryption efficiency.