Research On Key Technology To Prediction-based Dynamic Memory Allocation On Map/Reduce Platform

Map/Reduce big data processing platform, as one of the emerging big data processing technologies, is widely used in various fields. It is characterized by user-friendly programming model, data-local computation and high availability.Resource Management is one of the core functions of such platforms. One of the keys to efficient data processing is promoting the utilization of resource. Memory resource allocation of existing Map/Reduce big data platform is user-directed, which can hardly avoid the over-commitment of memory resource, leading to poor memory utilization in the platform.To this end, an online prediction based dynamic memory allocation technology is proposed and a prototype system is implemented in this paper. This technology can ensure the sufficient utilization of memory through dynamic reallocation of allocated but unused memory of tasks based on memory usage online prediction during tasks life cycles. Main contributions of this paper are:1) Map/Reduce-oriented hierarchical architecture model for dynamic memory allocation. The architecture model is constructed by prediction level, local dynamic allocation level and global dynamic allocation level. Working units between levels are mapped by one to many communications so that resource usage prediction and dynamic allocation can be coupled loosely and maintained easily. This model divides the dynamic resource allocation into local and global levels, which minish the risk of performance bottleneck under centralized decision and implementation of memory allocation.2) Map/Reduce-oriented online memory prediction technology. Memory usage charactoristics of typical Map/Reduce tasks are analysis and piecewise logarithm-like pattern of them is discovered. Based on the observation, liner regression and T-test methods for building memory prediction model by virtual of which the reallocation of memory can be calculated are proposed.3) Map/Reduce-oriented dynamic memory(re)allocation method. This method increase or release the allocation of memory based on prediction of memory reallocation. In case of insufficient of free memory when increasing allocation of certain tasks, a memory preemption strategy, considering the task run time elapse, task process and job process, is prepared to ensure the performance of running tasks and the fairness of reallocation.4) A prediction based dynamic memory allocation system Predra is implemented on open-source Map/Reduce platform Hadoop based on studies in this paper. Predra synthesizes prediction technology and dynamic memory(re)allocation strategy and is transparent to Map/Reduce applications, which makes it a portable system.5) Systematic performance evaluation is performed on Predra comparing with Hadoop, MROrchestrator and ADMP. The result shows that the reduction in average job turnaround time is 57% at most and 37% on average. The reduction in average job wait time is 68% at most and 49% on average.