Order Big Data Real-time Monitoring System Based On Storm

The era of big data,real-time and effective collection of mass order data,help the enterprise to obtain the target information intelligently from the data,can develop and improve the related business more effectively and systematically,The enterprise can carry on the two development and utilization to the massive data,further summarizes and processes the data,finally formulates the design plan which is more in line with the user demand,more suitable for the market.Storm is an efficient,safe and real-time data processing engine used by the system.The NoSQL database Elasticsearch and MongoDB can satisfy the mass data storage and query.In this paper,based on Storm,Kafka,NoSQL database,Elasticsearch and MongoDB,we design and implement a real-time monitoring system for large orders.The main data of the system is saved in Elasticsearch,and some configuration parameters and time stamp data are saved in MongoDB;The file system uses HDFS and the Scala language is used to write the code;The system uses the distributed message queue Kafka to connect the different functions of Storm in topology,which improves the reliability of the system.During data processing,the name of the operation,the time node,and the intermediate results are recorded in the system log,in order to solve system errors and improve system performance,the important results after treatment through the platform,Web pages show them in a variety of charts and tables,the display site has a search engine,through keywords and specific rules statement to search the target data,also support clicking the icon and label for quick search.The whole system is deployed in distributed cluster,which has the characteristics of high real-time,high efficiency,high fault tolerance and scalability,Data display site is powerful,the web interface is clear and concise,user experience is very good,The platform can monitor thechange of mass data in real time,from the structure of mixed and complex,large scale of data,by means of intelligence,dig out the valuable information,so as to create a certain economic and social value.