The Design And Implementation Of Graph Processing Middleware On Infosphere Streams

Many practical problems can be abstracted as graph models, such as networktopology, social network, web links and so on. In response to these needs, industry andacademia developed a number of graph-based algorithms, computational framework, opensource software library, the focus of these programs is to enhance the computationalefficiency and flexibility of graph, which in time to meet the demand of application. Butwith the expansion of these applications now and the new generation of applications anddistributed processing, streaming processing, the old solutions can not meet therequirements of the current application, so designing an efficient framework or tool ismeaningful.It is a large-scale graph processing middleware developed based on the IBM’s streamprocessing system(InfoSphere StreamS). The middleware uses StreamS part of the internalinterface and fault-tolerant data transmission, focus on the features and performance,simplifying the complexity of middleware. It take the vertex as the core, designedsubsidiary structure to speed up graph traversal and query, use pre-allocated memoryrecovery techniques to optimize the gentle figure of the perforance of basic operation. Thecore structure of the internal graph processing framework based on Groogle’s Pregelincreases the size of the data can be handledand performance, increased functionality andsize scalability. By caching, communications, parallel processing, on-line optimizationcalculations accordingly to meet the characteristics of stream processing applications.Considering the mapupdate, computing, and streaming query processing, different fromthe previousframework, the middleware functions to query the core to do the design,update and ensure StreamS applications, computing, stream processing. AccordingStreamSwrite interactive interface specification for the application running onStreamSprocessing middleware provides a standard chart, the internal implementation oftheapplication transparent. Tests show that the query performance on the core functionality of themiddleware(three working nodes) relative to the use of stand-alone general-purposelibrary to achieve the plan dealing with the traditional framework to improve queryperformanceby130%, compared to other multi-unit (three operating junction point)parallel graphprocessing framework to improve the average performance of14%. Themiddlewaresupports the dynamic update plans, calculations and queries, when the figurereachedthe edge of5.1million, the calculation of the impact on query performance of0.5%.When the map scale to800,000vertices,5,100,000edges, using a common libraryof stand-alone processing framework to achieve in the calculation of time, CPUutilizationreached100%, memory usage reaches98%, while each of the middlewareOperating junction of the average CPU utilization of69%,40%memory usage onaverage.