The Design And Application Of Real-time Stream Data Computing Model Based On Node-red And Redis

In the last few years,the rapid developments of new information technique and application modes such as cloud computing,Internet of Things and social media,are keeping pushing human society towards the age of Big Data.In the age of Big Data,the stream data is showing characteristics as real-timing,burstiness and disordering,which,put higher and stricter request to the stream data processing system.As the current real-time data processing systems are confronting difficulties of business expanding and data stream managing,this paper is focused on the solution of the above two problems by putting forward a new real-time stream data processing model based on flow management of Node-red and memory computing of Redis.This thesis designs the model from the overall structure,it redesigns the data input node,data output node,data computing node for Node-red and access node to Redis,the development of each node uses Node.js,which is an asynchronous programming language,and the communication between nodes is completed through pub and sub mechanism of Redis and socket.io of Node.js.Finally,this thesis resigns these nodes to Node-red to make it a complete real-time data stream computing model.In the process of real-time data computing,maximum and minimum values,cumulative sum and Top(n)are often seen,which base on the removing of repetition and then computing.This thesis comes up with the method of Redis based ordered set to remove repetition and recount,by analyzing Redis ordered set source code and combining with Skip List,and implements the application of this method in the real-time stream data processing model through the newly designed Redis database access node.One important task after the finishing of designing real-time stream data computing model is to apply and testify;therefore this thesis designs and achieves the web-site visiting monitoring system to apply and demonstrate the analyzing results on screens.This system includes three modes: real-time data acquisition module,real-time data analysis module and data visualization module.Among them,the real-time data analysis module is finished using the real-time stream data processing model,data visualization module is a web application with the use of Node.js express framework,what users need to do is simply logging in on the browser to access the monitoring page,while assembling the chart in data visualization module with Highcharts to adapt the diversity of data brought about by the expansion of business.This thesis also provides testing results and function analysis,which all meets the request.In short,the whole process from designing to the functioning of the real-time data stream computing model is covered in this thesis,feasibility and effectiveness in the actual production line has been verified at the same time its.