Font Size: a A A

A Scalable MapReduce For Multicore System

Posted on:2018-10-17Degree:MasterType:Thesis
Country:ChinaCandidate:Y F YuFull Text:PDF
GTID:2348330512986731Subject:Computer software and theory
Abstract/Summary:PDF Full Text Request
Parallel programming is an important way to make efficient use of multicore re-sources.However,traditional parallel programming techniques require the program-mer to manually manage many details,which makes parallel programming difficult and error-prone.MapReduce,a simple programming model,alleviates the burden of pro-grammers by providing automatic parallelism.Phoenix,a MapReduce library for mul-ticore,demonstrates that applications written with MapReduce framework can get com-petitive performance in comparison to those written with Pthreads.However,Phoenix scales worse on a 32-core system.This dissertation aims to analyze the key factors that affect Phoenix's scalability,and then proposes a scalable MapReduce library SMR.The main contributions of this dissertation are as follows:(1)We analyze some key factors that affect Phoenix's scalability and perfor-mance by experiment and performance analysis tools.Phoenix will eventually lead to poor scalability due to multithreaded contentions for the shared address space.And,there is a barrier between map and reduce phases,which limits the performance of Phoenix.(2)We use a novel thread model Sthread to design and implement a scalable MapReduce SMR.SMR can avoid multiple threads contentions on the shared address space to improve the scalability.And,SMR can implement efficient non-blocking pipeline communication between map and reduce phases to improve the performance.(3)We evaluate SMR and Phoenix on a 32-core system by running Phoenix's applications and examine the impact of Linux kernel optimization on the scalabil-ity and performance of multicore MapReduce libraries.Experimental results show that SMR can achieve better scalability and performance than Phoenix for five applica-tions.Specially,performance improvements range from 9.OX to 26.7X when the CPU core count is 32.Furthermore,experimental results show that the latest kernel version can effectively improve Phoenix's execution efficiency,while the impact on SMR is small.(4)In order to support the iterative MapReduce applications efficiently,this dissertation initially designs and implements an iterative MapReduce library iSMR for multicore.SMR need to repeatedly create and destroy the environment for each iteration,wasting a lot of time,so that it can not efficiently support iterative applica-tions.iSMR creates and destroys the environment only at the beginning and end of the processing,and it uses thread pool to reuse threads to improve the processing efficiency of iterative applications.The SMR presented in this dissertation provides programmers with a simple and efficient way to use multicore resources.In addition,the analysis of multicore MapRe-duce's scalability is great significance to the analysis of other Pthreads programs.For the analysis of Linux kernel optimization in recent years,it is of reference value for the related research work of Linux.
Keywords/Search Tags:MapReduce, Multicore, Scalability, Multiple Thread, Programming Model
PDF Full Text Request
Related items