The Design And Implementation Of Large Dynamic Graph Processing Platform

With the rapid development of “Big Data” and “Cloud Computing”,a new generation of information and communication technology has risen.It's too hard for intelligent transportation system,social network and other areas to meet the public demand with traditional internet technology.The question of how to deal with the problem of complicated graph structure behind these areas' has become a focus that more and more research institutions pay attention to.Under the environment of information expansion and complicated data,the processing technology of large dynamic graph is changing quickly.Google,Facebook and Apache represented by research institutions have pushed out a series of frame platform aiming at the storage,index and iterative processing of large dynamic graph to meet the demand of different backgrounds.At present,the processing frame of large dynamic graph mainly focuses on two models,which are MapReduce and BSP.The model of MapReduce mainly applies to batching of big data and its application programming interface is relatively mature as well as easy to achieve with high versatility and abstractness.However,it's unfit for iterative computations and those who have high real-time demand.The model of BSP,which is a state one,introduced a concept of “super step” that applies to iterative computations and matrix calculations,however,it set a high standard for each node of the memory performance in the cluster and the existing framework still needed to improve the stability.The processing platform of large dynamic graph will combine the advantages of Model MapReduce and Model BSP.It will aim at the high demand features of large dynamic graph's processing algorithm to data scale,iterative efficiency and timeliness,take the model of “MapReduce + BSP” to satisfy the demand of large amount of data calculation and multi-iterations based on the frame of Hadoop.It has made some modifications to the core source package of Hadoop on the basis of HDFS distributed file system,as well as increased the real-time monitoring function to check the dynamic change of graph file.At last,it developed a large dynamic graph processing platform,which is a lightweight framework platform,aiming specifically at large dynamic graph processing.The platform of large dynamic graph processing will retained the characteristic of old Hadoop frame,in addition,it will increase four main modules,which are iterative control,real-time monitoring,cache data acquisition at Map stage and local overflow write at Reduce stage.This thesis will also introduce the process of development from demand analysis of platform,overall design,detailed design and implement test.