The Design And Implementation Of Task Management Module And Data Analysis Module Soak System Based On Hadoop

With the development of Hadoop related technologies, the requirements of performance test of MapReduce, Hbase, etc are increasing. Intel invested CDH, the most popular Hadoop version in the world now, and they want to find performance problems of CDH before customers put it into use. There comes the reason of developing Soak tool. How to manage simulated customers’ App operations and analysis collected data is a problem in Soak.This paper focus on task management module and data analysis module’s design and realization. These modules solved above problems. Task management module focus on management about simulated customers’ App operations. These operations are processed on different nodes. Thus Soak use Client/Server architecture. Master Node is responsible for distributing tasks to different Agent nodes. Agent nodes focus on running these tasks. In this way, one Agent node can simulate multiply users operating multiply Apps.Data analysis module focus on analyzing collected performance data while running tasks. First, Soak need to collect the executing time of tasks. Second, Soak need to collect performance parameters of Agent nodes like CPU,10 and etc. Soak design a common data structure to store data and analysis data.Soak use Virtual User to simulate application operations and multiply threads are applied to manage tasks. What’s more, Akka helps message communication between master and agent nodes. Soak also use Dispatcher to analysis and transmit messages. While collecting performance parameters of Agent nodes, Soak use JRRD to read Ganglia’s data. Spring and Gauva are also applied in Soak.