Research And Implementation Of Real-time Computing System Based On Storm

Today’s world is the era of massive data. The daily operations of enterprise often generate TB levels of data, so how to good use of these data is a very critical issue. Map Reduce, HDFS and other related technologies help solving many difficulties of big data processing; however, they are mainly produced for off- line data processing. Now users more and more focus on the real- time processing of data. They also hope that all of the data has been processed into the pattern they hope to see for use after the process of data access. Obviously offline batch computing has been unable to meet this demand. Real-time monitoring, real-time recommendation and real-time reports and so on all need the technology. Users want to get the data within a short period of time and estimate the useful information, so as to fully reflect the values of the data. So the technology of big data real-time processing is particularly important.In this paper, the system is based on a open source and distributed processing system-Storm. It mainly combines with the Hive HQ L processing mode and implements a real-time computing system for big data analysis tasks, resolving the update delay problems of offline processing. This paper describes a specific process that contains the database access, the syntax semantic parsing of SQ L, logic execution plan, physical execution plan, Topology created and then submitted to Storm. The system also joins the Eclipse plug- in development technology and customizes the user interface to provide users with convenient. Finally, two types of service including real-time indicators 、 performance and time period statistics are selected from requirements description, implementation scheme to the cluster test to verify the system. Delay testing and fault tolerance were tested for the first type of business. According to the experimental results and practical applications, real-time computing system based on Storm in real- time and availability can meet the needs of users. Two types of business can normally calculate and the average delay is less than 600 ms. Even when a node goes down, it will continue to work, the average delay is not affected.The main results of this paper are outlined as follows: To achieve a real-timecomputing system and proposed based on open-source software—Storm. Users only need to write SQL; real-time computing tasks can be completed independently to facilitate the process. The innovation of this system is mainly reflected in two aspects: in function, this system can provide a real-time data processing platform fo r enterprise developers, one button to complete the entire business processes and improve efficiency. In technology, this open source real-time processing system replaces the traditional offline Hadoop to solve the drawbacks of slow data updates. Full use of the data value and reduce the losses due to delays caused by data analysis.