Design And Implementation Of Performance Analysis Tool For Big Data System

With the popularity of Mobile equipment, the development of sensor technology and the expansion of social networks, the data size has reached to an unprecedented level. The age of big data is coming. However, because big data has the character of massive and diversity, relational database is limited by expansibility and cost in the management of big data. Cloud Computing provides platform for manage big data with high scalability and high cost-effectiveness. So, more and more big data system has been transferred to the platform of cloud computing, such as Hadoop、Dryad、Pig、Hive etc. Among in these, hadoop is widely used. These systems free users from the complicated distributed computing. Users can only focus on the logic and flow of data processing. Technology like hadoop brings a lot of convenience for users, but they cannot see the underlying process of system clearly. They cannot effectively optimize performance of such systems. So it is necessary to develop a set of the automated performance analysis tool for the big data systems. This thesis discusses design and application of the performance analysis tool by author.The design and application of the performance analysis tool use Python and Java. The whole system is divided into three modules:PROFILER, OPTIMIZER and WHAT-IF ENGINE. Among these, PROFILER realizes the function of grabbing system performance index through BTrace tool. WHAT-IF ENGINE realizes the function of configure parameters based on the changing and the prediction of the performance of hadoop through a built mathematical model. OPTIMIZER realizes the optimization of the main performance parameters for hadoop platform automatically through Recursive Random Search Algorithm and WHAT-IF ENGINE module. The author completed the design and implementation of profiler module and optimizer module, and the implementation of WHAT-IF ENGINE module independently, participated in the creation of mathematics model of WHAT-IF ENGINE and the functional test of this performance analysis tool. Finally, the thesis describes some test sections from the author’s design, and shows the executive results of this system.At present, the development and test of this tool have been completed. The test results meet the requirement of the tool.