The Research Of Resources And User Behavior Testing Technology Based On The Hadoop Big Data Platform

Cloud computing represents the development trend in the IT field turns to intensive, large-scale and specialization, is the profound revolution going through in the IT field. It improves the efficiency of the use of all kinds of resources greatly, but also brings great shocks and challenges to the asset security and privacy protection of user information. As an open source basis cloud computing framework, Hadoop gets more and more attention in the business community, but the issues of cloud computing platform security has not been effectively resolved yet. Therefore, the weakness in Hadoop security mechanisms has become one of the main problems which hinder its development. So, the Hadoop platform security monitoring technology, to a certain extent, will enhance the users’ trust in Hadoop, thus promote the development of Hadoop.In this paper, based on the research of Hadoop security issues, aiming at the difference of Hadoop cluster data security and resource security, we proposed a data access anomaly detection technology of user behavior and a resource consumption anomaly detection technology, to monitor the security of Hadoop cluster, thus improve the security of the Hadoop cluster. The main work of this paper is as follow: First of all, according to Hadoop HDFS data access mechanism, we propose a user access behavior anomaly detection technology based on a hidden Markov model, which is different from the others based on correlation analysis. This method aims at a single user’s micro command sequence, avoids data mining process during correlation analysis and pretreatment process of the multiple feature data, which reduces the time and space complexity greatly, improves the performance of real-time monitoring to user’s abnormal behavior.Secondly, according to the distributed storage and distributed computing of resource consumption characteristic on Hadoop platform, we propose an anomaly detection technology based on the time series of resource consumption, with the k-nearest neighbor method applying to the time series model, combining with the sliding window, realized the subsidiary sequence pattern of local anomaly detection, thus found out patterns of anomaly. This method improves the general anomaly detection technology based on burst point, thus find out the slow changes, to achieve the detection of abnormal subsidiary sequence patterns.Through the research, the user data access security will be further protected, while the Hadoop cluster resource safety monitoring ability makes certain progress, and provide an effective reference for the goal of providing reliable service performance to Hadoop users.