Research Of Gpu Cluster-Based Mapreduce Programming Model

Nowadays, we live in the era of information explosion, and there are thousands or millions of data ceils produced every second. The data size grows in an exponential manner, and there are lots of problems and troubles when we process and analyze large data sets. How can we process these data sets quickly and efficiently? MapReduce is an efficient programming model, which was firstly proposed by Google, to deal with large data sets, and normally used for distributed computing on a great many of commerce machines. Therefore, recently, many efforts have been put to accelerate MapReduce model using various parallel architectures.Compared with commodity CPU, Graphic Processor Units (GPU) has a special hardware architecture, which provides higher computing performance and memory bandwidth. Meanwhile, with the development of general purpose GPU computing languages, GPU computing is becoming increasingly popular in various domains. More and more traditional applications can be easily developed on GPU, and obtain dozens or even hundreds of performance improvement. So there have been lots of efforts to implement MapReduce model on commodity GPU-based system. However, most of these implementations can only work on a single GPU or multi-GPU in one node, and they couldn’t utilize the power of High Performance Computing Cluster and providing large-scale data sets processing.In this paper, we present GCMR, a new implementation to accelerate MapReduce for large-scale data processing on GPU clusters. We have used Compute Unified Device Architectures (CUDA) and MPI parallel programming models to implement this framework. To derive an efficient mapping onto GPU clusters, we introduce a two-level parallelization approach: the inter node level and intra node level parallelization. Furthermore in order to improve the overall MapReduce efficiency, a multithreading scheme is used to overlap the communication and computation on a multi-GPU node. Compared to previous GPU-based MapReduce implementations, our implementation achieve speedups up to2.6on a single node and up to9.1on4nodes for processing small datasets. It also shows very good scalability for processing large-scale datasets on the cluster system.