Research And Implementation Of Set-oriented MapPartition Computational Model

With the development of Internet technology and the emergence of cloudcomputing technology, a large growing number of companies begin to face the massivedata processing. The traditional centralized or distributed approach is difficult to meetthe computing needs of large amounts of data, but the cloud computing technologyprovides a good platform for the large-scale data processing. However there is a hugedifference between the sequential program and the data processing under cloudcomputing environment, which has a high degree of parallelism, and is distributed. Howto provide users with a simple yet powerful programming model and interface, is thefocus of the study of cloud computing technology. Without powerful support foriterative programming models, existing cloud computing environments often requireusers to define some specific custom functions, at the cost of lack of control logic.In this paper, a Set-oriented Map Partition Computational Model (SOMP) isdesigned based on the research of existing computational models. The model consists ofa main function and some user-defined operations. In the main function, users canperform iterative programming as easy as sequential program, and user-definedoperations can be executed on each tuple in a set of key-value pairs by certainprimitives. Due to the delay execution and optimization of operations, SOMP model canmake full use of data caching, resulting in high efficiency. After the actual test, thismodel demonstrate a high efficiency in dealing with iterative calculation of data.