Design And Implementation Of MapReduce Cluster Of GPU Acceleration

Processing massive data in even faster speed is the eternal pursuit for datacenter computing. As the data is growing in increadble speed, and the desire of timeliness is raised up in some, the pressure of data-prcessing is more and more serious. Hencs, it had to take messure to update the software and hardware architecture and infrastrature. MapReduce is a distributed and parallel computing model. It is being widly used in domains such as enterprise datacenter-computing. These days, researchers embark on mining potential of this model. Among these solutions, hardware-accelerated MapReduce is a novel idea. In this paper, we will present a new MapReduce implements based on Graphic Processing Unit (GPU) acceleration. GPU is a kind of highly parallel many-core processor, which can lauch hundreds even thousands threads concurrently. In the high performance computing, heterogeneous coprocessors have gained wildly recognize. We try to combine the huge compute power with the advantage of MapReduce in data-intensive applications, and implement a GPU-acceleration-based high performace MapReduce cluster.The subject of this paper start the research work according to this motivation, the main work and achievements is as follows:1. We design and implement a GPU-Accelerated MapReduce Framwork, GAMR cluster;2. A GPU-based parallel sorting algorithm is presented, which is applied to GAMR system, accelerating the sorting about3to8times;3. We have analyzed the dataflow of MapReduce in detail, and we presented a formalized quantitativeperfprmance model. Which enable the performance evaluation canbe gained by formula calculating.4. We present a conjugation-gradient-method-based automatic turning method for the performance optimization of MapReduce cluster.The core motivation of our work is to extend the parallelism of MapReduce model, from multi-machine alone to multi-machine with many-core. According to our test, comparing with some other MapReduce implements, the MapReduce jobs running on GAMR cluster gain about5X speedup.