The Security Mechanism Design For Hadoop

In recent years, with the explosive growth of information, the era of big data is coming.Hadoop,as the technology of mass data storage and computing, gradually become hot Applications, Hadoop is an open source distributed computing platform of the Apache software foundation.Compared with traditional methods by buying expensive, high-performance large server for handling large amounts of data, Hadoop is more cost-effective, because it can be build and run on cheap business machine cluste for large data storage and computing through distributed data storage and computing framework. Because Hadoop itself has high reliability, high scalability, high efficiency, high fault tolerance, etc, it has gradually become the research focus of large enterprises and research institutions.But with the enlargement of the Hadoop Applications, its security problem is increasingly protruding, the unsafe factors of Hadoop in data storage and data processing technology turned off many enterprises who need to use it.This paper analysis the authentication and authorization mechanism for the HDFS and YARN, the core components of Hadoop2.0.Through a detailed analysis for authentication mechanism current adopted based on Kerberos and token based on asymmetric encryption,transmission encryption mechanism,access control mechanisms based on Linux/Unix system in the process of service,we sum up the disadvantage of these safety measures in the aspects of safety and efficiency and puts forward a new security framework on this basis.In this paper, the innovation points:1. This paper puts forward digital certificate authentication technology based on PKI take the place of original authentication based on Kerberos,and design safer, more effective authentication Token on the basis of the principle of asymmetric encryption.2. In this paper,we introduce AES symmetric encryption technology based on the digital envelope to the process of secure transport encryption in the Hadoop2.0.Then we can assure the security and efficiency in the process of data transmission.3. In view of the service characteristics, multi-user, big data, multi-threaded service for Hadoop,this paper puts forward a more flexible, more rigorous independent access control mechanism based on roles, and use the ACL technology to implement. Compared to the original simple authorization management, the new authorization management this article proposed costs less, judge permissions more concise, manage authorization more closely and be more adaptable to Hadoop2.0 security policy.