MapReduce Development Method For Data Transformation Based On Model Transformation

In the era of Big Data,MapReduce,a famous parallel computing model,has been widely applied to Big Data processing and transformation in both academia and indus-try.However,developing a MapReduce program is not as easy as it looks because developers must understand the requirement of the data processing,the concept of par-allel computing and APIs of MapReduce framework(e.g.Spark).For the hard programming of MapReduce,there are lots of studies working on ease of parallel programming,and many solutions have been proposed.However,those so-lutions usually focus on low level improvement,which cannot reduce the gap between different parallel platform and programming details.Model transformation technology is proposed by MDA to process model instances and generate new model instances,which can well extract the logic of model processing and avoid the detail in model transforming.Therefore,there is a good exploration to apply model transformation into the MapReduce development for data transformation.In this dissertation,we mainly explore how MDE can facilitate MapReduce de-velopment by proposing a model-based approach to MapReduce-based Big Data trans-formation,including the following aspects:1.We propose a MapReduce development method for data transformation based on model transformation.Our approach enables developers who have little knowledge of parallel computing and MapReduce APIs to specify a data transformation as a set of declarative QVT relations.Afterwards,a MapReduce program is automatically generated from these QVR relations.2.To facilitate the code generation,the dissertation define midCore,an abstract model of existing MapReduce frameworks(e.g.,Spark),as an intermediate layer between QVT and MapReduce programs.In addition,we define a set of transformation rules to transform from QVTr to midCore and from midCore to Spark.3.We implement a prototype tool,namely,QE2S,that realizes the conversion from QVT to midCore and the conversion from midCore to Spark.4.Finally,case studies show that our approach can effectively accomplish common data transformation tasks with consume no more than 30%time of normal devel-opment.