The Design And Implementation Of Hadoop Management Software Data Hub

As the age of big data comes, Hadoop is widely used not only in big Internet enterprises, but also in big data projects of other industries, such as traffic management, bill inquiry and video monitoring. Thus, quickly constructing a well-functioning Hadoop cluster, offering visual operation, administration and monitoring becomes a basic step of constructing a big data project.Open source makes Hadoop vibrant, but also makes it lack of maturity, haleness, usability and powerful technical support. It seems difficult to build a compatible, efficient and stable Hadoop cluster. Thus, we develop a Hadoop management software Data Hub on B/S design pattern.The Data Hub adopts GWT (google web toolkit) as its basic architecture. It combines RPC and PUPPET for communication and synchronization. The system consists of four modules:RPC module, mainly responsible for message transferring and remote procedure call between client and server; installation and operation module, mainly responsible for installing Hadoop components in a cluster, starts and stops services; state monitoring module, mainly responsible for the real-time monitoring for Hadoop cluster state, report the health status of each service and equipment; system configuration module, mainly for custom configuration for each Hadoop component, make the configuration item persistent to the local and use PUPPET to synchronize in the cluster.With the Data Hub, corporate users with big data requirements can easily deploy a Hadoop cluster and configure the cluster. At the same time, Administrator can monitor the cluster in a visual way and find problems of the Hadoop cluster in time.