Design And Implementation Of Real-time Big Data Processing Platform Supproting Workflow

In recent years,big data has penetrated into all aspects of human life.With the development of mobile Internet and Internet of Things,China's big data industry has entered an explosive period,and big data has risen to the national strategy.Large data processing systems also have more diverse business needs,leading to the gradual maturity of various technologies supporting large data processing.Integrating different technologies into a large data processing system,this system architecture has been widely used in practical production.But its learning cost and maintenance cost are very high,so it is difficult for the same developer to maintain and expand the system which is composed of so many complex technologies.In view of the above problems,combined with the project of the research group,this paper designs and implements a real-time large data processing system supporting workflow.Firstly,this paper studies the organizational structure and underlying implementation principle of mxGraph framework,and combines it with Web technology to build a large data management platform supporting workflow.On this basis,the configuration of Flume,Kafka and the working principle of Park Streaming are summarized,and the large data processing system is deployed.Finally,a real-time large data processing system supporting workflow is realized by combining the visual information management platform with the large data processing system.The core function of the system design is to realize the visual configuration of large data processing through the large data management platform supporting workflow.The configuration information of the big data processing system is extracted and handed over to the big data management platform for unified configuration management.Large data management platform uses XML format to transmit configuration information.After verification,configuration information can be read and written in the database or written in the configuration file of large data processing system.Through visual configuration,the workflow is organized into a large data processing flow that conforms to logic and rules,and the large data processing system is controlled to implement large data processing in a proper way in the computer.Developers only need to learn the principle of large data processing technology,master the configuration parameters of large data processing technology platform and simple data flow direction,so they can easily realize real-time data processing business.This paper solves the difficulty of developing the system and the high maintenance cost.At the same time,the system has good expansibility,and real-time feedback information can be provided for every step of every large data processing system.