The Design And Implementation Of Task Module In Linux Cluster Data Generation And Management Tool

With the rapid development of of Internet and explosive growth of data, fast data processing for meaningful information becomes the critical point.It’s difficult for the information system of traditional architecture.which is expensive.to deal with these problems.Hadoop supports a stable system composed of thousands of cheap machines,used for petabytes large data storage and computing.which makes it receive extensive attention of domestic companies,more and more Proof of Concept projects for hadoop are in progress, but we find that POC needs a lot of shell script and java program.but there are no management and monitoring for those tasks in POC projects.This paper compares several distributed software such as ganglia、nmon、saltstack and hadoop,with the analysis of their applicable fields,I find that they are not very good to solve these problems.In order to solve following problems.collecting hadoop related data, test data generation,distribution and execution of tasks.task monitoring.this paper comes up with a distributed software called linux cluster data generation and management tool.It is a lightweight software deployed in a linux cluser.it provides these functions:cluster monitor, data generation、execution and management of distributed tasks.which makes it to be complementary to hadoop products.This paper researches、designs and implements task module in linux cluster data generation and management tool,which is programmed in java,supported by java NIO、 java Thread technologies.The tool also supports parent classes for task and scheduling strategy users can write code with execution sequence and strategy logic by extending these parent classes.and this tool is responsible for distribution、execution、monitoring and management.The tool makes it easy for developers write sequential logic code、log information collection code without having knowledge of distributed programming.People who are good at distributed programming can also reuse distributed framework and task scheduling logic provided by the tools,and extend the function of the tools by these code.