Stream Computing Service System Of Smart City's Big Data Platform

There are many computation processing requirements for stream data in smart city big data processing application,such as instant message processing and public opinion monitoring in smart neighbor communication application,building state information monitoring in smart building application,city traffic path planning in Telematics application.These smart city big data processing oriented stream computing applications often adopt cloud infrastructure and distributed architecture.It requires the application developer skillful as these stream computing applications involve data analysis processing,complex event processing and business rule processing.Thus it's a real applying problem that how to provide a general platform service for smart city oriented stream computing application's development and deployment based on cloud infrastructure.To solve these problems,this paper proposes a stream computing service model based on which we design and implement a stream computing service system using storm,on the basis of studying related technology like stream computing,complex event processing,rule engine,considering the requirements of stream data processing,complex event processing and business rule processing,based on the "smart neighbor community service software system development" project of our laboratory.This system solves the problem in Storm that the executing entity's parallelism of application couldn't be changed dynamically and could provide platform service for stream computing applications.Developers only need to define SCS application description file and develop related business logic to build a stream computing application.And based on this system,we developed instant message module in neigh communication service.Tests and applications show that this system is usable and effective.This paper's main work includes the following:1)Stream computing service system is based on SCS model and open source framework Storm.But the executing entity's parallelism couldn't be changed dynamically,which causes the SCS application hard to scale out when data amount is high.This paper proposes an algorithm called PDC to dynamically change the executing entity's parallelism of Storm application.And based on this algorithm designed a module called PDCM.Test and application show that this module could dynamically change the executing entity's parallelism of Storm application and the relationship of different executing entity's parallelism.2)Stream computing applications that are based on cloud infrastructure and distributed architecture require developer to be skillful.In order to support the development of stream computing application on the basis of platform service,this paper proposes a general SCS model.Designs and implements a SCS model definition tool based on this model.This enables develop be able to build an application that could run on SCS system by simply define SCS application description file and implement key processing logic.Test and application show that this tool could correctly build stream computing application.3)Stream computing applications could be defined by SCS application description file and be built based on SCS model.The execution of application requires SCS executing engine.This paper designs and implements SCS executing engine based on open source framworks Storm,Esper and Drools.This engine makes complex event processing that based on Esper and business rule processing that based on Drools as uniform processing node.And it solves the scaling problem by using the distributed mechanism of Storm.This engine also implemented the data dispatching mechanism between nodes that have parallelism and splitting relationship.Test and application show that SCS executing engine is feasible and effective.