Design And Implementation Of Real-time Processing System Based On Storm

With the development of the Internet, cloud computing, mobile networks and Internet of things is growing rapidly, big data gradually become hot in recent years. Big data processing with our usual life more and more closely, such as the Internet, mobile, wireless, etc., are to deal with global users of interactive data. Hadoop has achieved good results in big data processing, but it is limited to batch processing for offline big data analysis, which can not be well responded to real-time big data processing. However, in real life, often need to analyze and process real-time data, and can get timely feedback. At this point, the emergence of Storm to solve this problem, making real-time large-scale data processing can be more easily achieved, to make up for big data processing in real-time requirements.In this context, this paper designs and implements a real-time data processing system based on Storm.Firstly, this paper presents a real-time processing architecture of big data, which divides the real-time processing system into five layers: data acquisition and preprocessing, data access layer, real-time processing layer, storage layer and application layer. Combining Kafka, Storm,Redis and Web achieve a complete real-time processing architecture. The complete system architecture has laid the foundation for the construction of perfect real-time processing system.Then, this article focuses on the application layer deployment in the real-time processing architecture. The application layer of the system is divided into client, Node server, application server and application storage, and the front and back separation is realized, and the interaction between the data is standardized. The application of the real-time processing system is provided to provide the infrastructure. At the same time based on the architecture design and implementation of the real-time processing system prototype, the system includes data access module,real-time processing module, the results show and other modules, and use Redis and MongoDB to store data. One-click completed the entire business process, improve efficiency.At last, through the functional test, the performance test and the horizontal expansion test, the real-time processing system based on Storm is more superior to the traditional data analysis system in real-time,usability and scalability.