Research And Application Of Parallel Computation Framework Base On Task Type

The coming of big data era makes it difficult for traditional stand-alone machine to solve huge data processing problems.Distributed computing is becoming more and more important.Distributed computing is mainly divided into data-intensive computing and high-performance computing.In order to solve data-intensive computing,industry proposes various frameworks base on MapReduce model.However,the main parallelization of high-performance clusters still adopts MPI with low level of abstraction.This paper improves the DCR distributed computing framework in order to reduce the difficulty of developing parallel program on high performance computing clusters.The framework helps programers get rid of paying attention to the detail of parallel design by providing distributed features and supporting various computing models.The framework can take advantage of share-memory cluster to implement distributed computing system.The computing models of this paper are de-compose,compute and reduce.This paper proposes and implements an iterative computing model and a real-time computing model base on last generation system.It introduces the task type mechanism which enriches the schedule and parallelization strategies.The framework take advantage of data-locality by introducing cache model.By introducing Reactor pattern,the framework gains a more efficient way for network communication.Finally,three applications are parallelized by this distributed computing framework.They present and verify the usge of iterative model,task type,cache model and so on.The result demonstrates that DCR can support the parallelization of these kind of computing model and provide a linear-approaching acceleration effect.