Design And Implementation Of Hadoop Cluster Management And Charging System Based On Web

During the internship in Big Data Research and Development Department Sugon Information Industry Co. Ltd.(Beijing), the author joined a project named Hadoop Cluster Management and Charging System based on Web. The system can make the management of the Hadoop Cluster unified, scientific and visual, and simplify the maintenance of engineers, it also have charging function for the cloud services of companies.The author’s major work could be summarized as follows.(1) In the system requirement analysis phase, the author participates in the system requirements discussion and analysis, and understands the system’s functional and nonfunctional requirements.(2) During the design and implement phase, the author completed the design and development of the seven modules which include:1) Hadoop components information collect module:the system collect the information of Hadoop components circularly by different methods, and also has alarm function;2) Cluster status collect module:the system collect the hardware resource information of each node of the cluster circularly,, and also has alarm function.;3) Job information collect module:the system analysis the Hadoop job log file and access the job information circularly;4) Job statistics and analysis module:the system statistics the job informationand calculate the user expense, then analysis the job information with MapReduce method;5) Hadoop components installation module:according to the requirement of user, the system run some commands of Linux to accomplish the installation of Hadoop and the start of components;6) Hadoop components configuration module:the system modifies the XML configuration file with the information from Web;7) User management module:the basic information, role information and expense information is management by the system;(3) During the system testing phase, the author builds test environment, writes the test cases of core modules and take part in the functional test. Finally, Hadoop Cluster Management and Charging System based on Web works well on job collection, statistics and analysis, realize the visual, unified and scientific management of Hadoop Cluster, and have the function of management and charging.