| With the rapid development of applications in domains such as social networks,recommendation systems,biological information networks and web pages,a large number of data are modeled as graph to mine valuable information.In order to satisfy the processing requirement of graph applications for large-scale graph with hundreds of millions of vertex level,the distributed graph processing system can significantly improve the processing capability of large-scale graph with the help of the massive storage capacity and the powerful distributed parallel processing capability of the cloud environment.However,due to the existence of the synchronization phase of the current popular distributed graph processing system based on BSP computing mode,the “bucket effect” has a great impact on the performance of the upper level graph application.How to further optimize the performance of the graph application on the basis of the existing system,so as to realize the fast and efficient processing of large-scale graph,is a hot issue in the field of graph computing.Based on the characteristics of the elastic resource supply of cloud environment,this paper aims at starting from the perspective of the allocation of underlying resources in the cloud environment,to allocate the appropriate resource for the sub tasks of a given large scale graph processing problems according to the execution relationship between them,and to study the resource optimization mechanism for the large graph processing.The performance constraint caused by the “bucket effect” is alleviated under the condition of ensuring the utilization rate for the underlying resources in the end.The main work includes the following three aspects:Firstly,this paper studies the extraction and analysis of large-scale graph application.Large-scale graph application has the dynamic and diverse resource characterists because of the differences of algorithms and data structure and the inherent pattern of large-scale graph application in the distributed environment.To this end,this paper makes a secondary development of the existing open source large-scale graph processing system and extracts the execution characteristics.Furthermore,this paper analyzes the execution mode of specific graph application in the corresponding graph data structure and provides the research foundation for the subsequent underlying resource allocation.Secondly,this paper proposes the application-aware resource allocation and dynamic adjustment mechanism.Taking the resource requirement characterists and the process of resource allocation in the cloud environment into consideration,this paper designs a two-stage resource allocation method to allocate the resource accurately to reduce the impact of the “bucket effect”,thus improving the performance of the application layer and the efficiency of the overall resource utilization.Finally,based on the existing open source large-scale graph processing system Giraph,this paper designs and implements a set of application-aware large-scale graph processing system,which integrates the corresponding functional modules,and deploys this system in the cloud computing platform of Southeast University to carry out the graph processing.On the one hand,it provides support for the analysis of the application execution pattern.On the other hand,it is used to verify the research ideas and practical results of this paper.Overall,based on the experimental results,the proposed execution feature extraction and execution pattern analysis mechanism can accurately predict the change trend in a segment of the application,and the proposed application-aware resource allocation and dynamic adjustment mechanism can effectively alleviate the impact of “bucket effect” on application performance,and significantly improve the utilization efficiency of resource.This paper provides a new solution for further optimizing the performance of the graph processing on the basis of the existing system. |
PDF Full Text Request