Research And Realization Of Monitoring System Of Big Data Platform

As the era of big data comes,more and more enterprises have established applications with high performance and expansibility based on Hadoop cluster ecological platform to store,analyze and process their mass data.Thus the monitoring of big data platform has become a hot issue.Though there are monitoring and management tools for cluster in industry nowadays,many problems remain to be solved,especially the one that there is no mature access control strategy on the multi-tenant condition,which have potential risks in security of sensitive data in cluster.In addition,the requirement for service version in cluster performance is too high and there is not a complete platform to combine cluster monitoring and multi-tenant access control and management.In accordance with above issues,this thesis describes a big data platform monitoring and multi-tenant access control and management system.The research of this thesis is carried out from following aspects:firstly,aiming at the problem that cluster monitoring software restricts service component version excessively,a cluster monitoring solution based on Ambari is proposed,which can monitor the node health in Hadoop ecosystem and the health of service components deployed on nodes.Thus,to complete the monitoring of nodes and service components and send alarm when they have fault.Secondly,aiming at the situation that the identity authentication and access control of the cluster data access in the multi-tenant environment are relatively independent,this paper proposes a multi-tenant management scheme for cluster access.Combining the Kerberos authentication protocol with the Sentry access control model,combining with the specific needs,Access control model to optimize and improve the definition of a set of role-based access control model to ensure the security of multi-tenant access to the cluster data to protect the access control cluster components security to achieve a cluster-level granularity safety.After testing,the role-based access control model can guarantee the security and privacy of cluster data,and realize the multi-tenancy management of big data platform.Thirdly,based on the two functions of cluster monitoring and trunking multi-tenancy management,the Vue.js front-end framework is used to integrate the monitoring and management of the cluster into a unified platform.The front page visualization page of the one-page web application is constructed.The cluster nodes monitoring information and multi-tenant management processes are presented on a visualization platform for easy administrator operation and processing.Based on above research and results,the thesis establishes and realizes the monitoring system of big data platform which enables administrator to monitor the status of cluster clearly and directly while ensures that the access to cluster data is on the basis of role-based access control,thus achieving the security and usability of big data platform.