Research And Implementation Of Virtual Machine Management System For Large Scale Graph Processing

The data has a trend of being a graph due to its complicated relationships and dependencies. The Pregel system that is proposed by Google brings new idea towards large-scale graph processing by adopting the iterative BSP model. However, there are still some problems concerning the current research work. On one hand, they ignore the performance degradation or resource waste due to poor isolation between applications, on the other hand, existing research work ignore the various resource requirements when the application is in different period. In order to address these problems, this thesis proposes the application-aware graph partition and resource scheduling mechanism based on the execution pattern in order to boost up the performance.Firstly, this thesis analyzes the extracts the execution pattern of the large-scale graph processing framework. Based on Pregel-like system, the thesis analyzes the vertex update function, extracts the execution pattern and formulates the relationship between resource consumption and execution pattern. The execution pattern is the basis of the following work.Secondly, this thesis proposes the application-aware large scale graph partition mechanism. Focusing on the execution pattern, this thesis partitions the graph wisely to balance the computation load while reduce the network communication overhead so as to better allocate and schedule resource as well as raising the performance of grapg processing applications.Thirdly, this thesis focuses on the application-aware virtual resource allocation and scheduling. Based on the relationship between resource consumption and execution pattern, this thesis's mechanism utilizes the execution pattern from upper layer application to implement a fine-grained resource allocation and scheduling, in order to boot up the performance of the application while the resource utilization is guaranteed.Eventually, based on the Openstack virtualization environment, this thesis designs and implements the graph processing system nutcat with the application extraction module, the application-aware graph partition module and application-aware virtual resource allocation and scheduling module. The system is deployed in the real productive environment, the SEU CLOUD.Experiments in real environment turn out, the graph partition mechanism and resource allocation/scheduling proposed by this thesis could largely increase the performance of those large scale graph processing systems while guaranteeing the resource utilization.