Research And Implementation Of Stream Data Processing Platform Service Based On Container Cloud

Stream data processing and complex event processing is widely used in real-time analysis of financial big data,Internet and Internet of Things.Constructing a stream processing application in cloud architecture often requires developers to do a lot of configuration things and environment setup.It brings a lot of difficulties for the development and maintenance.Though,there are needs to help developers develop stream processing applications more easily.In order to solve the problems described above,in this paper,a Stream Data Processing Platform Service(SDPPS)based on container cloud is proposed.The system includes several modules: stream application developing module,platform service management module,stream application runtime module,stream platform service monitoring module and a load-aware scaling module.Through experiment and running instance,the SDPPS system is proved to be effective and feasible.The main contributions of this paper are as follows:1)Application development mechanism on SDPPS.Stream processing application development includes a lot of environment setup and configurations this can slow down the development of the application.In this paper,an application model based on container cloud is proposed.Application developers can design application and generate describing document of the application.Based on the generated document,the runtime engine starts the related services and application instance.Runtime instance shows the application development mechanism is effective.2)Status monitoring of SDPPS running instance.Monitoring the application and service status is important for developers,but getting the monitoring information of various SDPPS running instances are complex due to separated ways for retrieving.This makes the maintenance to be difficult for developers.In SDPPS different through proactive and passive monitoring mechanisms,different service status monitoring are unified.SDPPS offers a general user interface for status monitoring.Experiment shows the impact of monitoring module is less than 1% on CPU performance and less than 5% on memory usage.3)A scaling mechanism based on prediction of data flow.Stream applications may meet different data flow rate when running.Application throughput may fall when it fails to scale when meets a large incoming data volume.To prevent this from happening,in SDPPS we proposed a scaling mechanized based on prediction of data flow using LMS algorithm.The scaling module scales instance parallelism based on prediction result.Experiment shows the NMSE measurement of prediction algorithm is 0.51 and the scaling mechanism can effectively adjust the instance scale after prediction.