Design And Implementation Of Data Stream Query And Analysis System Based On STORM

Data stream analysis has always been the focus of research,especially the development of big data in recent ten years,which makes data stream analysis more and more important and popular.Based on the data stream analysis system of laboratory project,this system is a general data stream analysis system based on storm.So far,the number of available data flow analysis system is still relatively small and it's difficult to get started.It requires relevant professionals to do so.Aimed at the above problems,by analyzing the needs of data stream development,a SQL like data flow stream language scql is proposed and implemented,and the generated logic applications are deployed to storm.This system is simple and easy to use,as long as you understand SQL and simply learn scql syntax,you can use it,and you don't need to manage a lot of configuration information of storm.A large number of tests show that the system is feasible and effective.The overall system is divided into three module including basic module,adapter module,compile module.Basic module is used to provide data processing,the adapter module is used to deploy applications to the storm,the compiler will split and merge operator module which is used to optimize the scql statement after the syntax analysis,semantic analysis,operator,finally compiled to build a logical application.Syntax analysis is to extract the information of leaf node from the abstract syntax tree.Semantic analysis is to redvelop the result of syntax analysis and then to create the information of table and the description of expression.In the next step,we will split operator according to the statement actuator,and then generate the physical implementation plan.To sum up,this paper introduces the data stream background and related technical analysis.On this basis,analyzes the requirements of the system,and then proposes the overall architecture and design proposal.In the core part of the paper,the design and implementation of the system are discussed in detail,and given the test cases.