An Inspection Tool For Transwarp Data Hub:Design And Implementation

In recent years,as the development of big data technology has become more mature,big data has long existed not only as a concept,and various industries have begun to apply big data technology to their own enterprise production environments.Since the release of TDH(Transwarp Data Hub),after years of development,its product system has been continuously improved and its performance has been continuously improved.Compared with open source Hadoop,it has more centralized management of the cluster,avoiding many problems of cluster management and upgrade.However,even so,its management of the cluster still cannot cover all aspects of the cluster.For large-scale clusters,its security issues involve too many aspects of data.It is difficult for inexperienced operation and maintenance personnel to accurately find the information they need.Therefore,in order to strengthen the TDH big data platform's ability to manage the cluster,it is of great significance to implement an inspection tool that helps it obtain secure data in all aspects of the cluster.The inspection of the cluster by the inspection tool mainly includes five modules,namely,checking the basic information of the cluster,checking the HDFS information,checking the node information,checking the service information,and checking the data table information.The basic cluster information mainly includes inspection error information,cluster alarms,various version information,and node load indicators that occur during the inspection process.HDFS information includes basic HDFS usage information,HDFS file space information,fsimage merge status,and HDFS parameter configuration.The node information mainly includes the basic environment of each node in the cluster,the disk information of each node,the network status of each node,and the port connection.The service information mainly includes the running status of the services in the cluster,the role information of the services,the startup time of the services,the disk configuration and process information of the services,and so on.The check of the data table is mainly for the TEXT table,ORC table,HBASE table and ES table in the cluster.The inspection tool analyzes the reasonableness of the various types of information collected according to the inspection rules of each module summarized in advance,and writes the cluster security data and cluster abnormal alarm information into the inspection report.The inspection tool is a design and implementation of a management platform that can be integrated into TDH by combining the actual needs of the customer's operation and maintenance personnel and the company's internal technical support personnel with TDH and the best practices of cluster indicators in the actual production environment.Big data cluster security inspection tool in Transwarp Manager.As far as the clusters used in the test are concerned,each inspection tool can complete the inspection of the cluster within three minutes and generate inspection reports in the form of Excel,HTML and JSON.It has been applied to customers' actual production environments and can be very good.Help operation and maintenance personnel obtain security data in all aspects of the cluster.