Design And Implementation Of Real-time Computing Task Management Platform Based On SSM

With the rise of the mobile Internet,the data information faced by people has begun to increase in a geometric trend.Many companies have discovered the commercial value of it,and want to mine useful information from massive data to support business activities.Data analysts and enterprise managers tend to obtain real-time data changes.However,in the face of large-scale streaming data,conventional analysis methods and analysis frameworks are inefficient in resource use,calculation results are not timely and accurate,and development cycles are long.Therefore,a different computing platform based on Spark and Storm is needed to promote the efficient operation of enterprise data services.The high throughput and low latency characteristics of Flink computing framework when processing large-scale data and good support for streaming data make it the preferred framework for real-time computing platforms.This article uses Spring and Spring on the basis of Flink computing framework MVC and My Batis built a web terminal,further encapsulated the design of Flink's Table API,simplified the configuration operation and use of real-time computing tasks,and provided an effective solution for enterprise data transfer.In the design and development process of the project,the author first participated in the platform's needs analysis and feasibility analysis.Based on the needs analysis,the four functional modules of the platform were designed and implemented respectively:(1)Source management module: including Kafka cluster management,Kafka Topic management,HBase dimension table schema management.(2)Sink management module: supports registration of Kafka Topic,Hive table,Ti DB table,and Hbase table to the platform as a sink data source.Support batch import of topics through Excel templates,and online creation of data tables through SQL statements.(3)Task management module: This module is used to add,delete,and modify real-time calculation tasks,and start and stop calculation tasks through the Rest API.(4)Task monitoring module: This module is used to monitor the running status of real-time computing tasks in the platform.You can set monitoring rules on the running status and running time of the tasks,view the task running logs,view the current data throughput of the cluster,and view the tasks delay.Finally,a black box test was performed on the system.The test results show that on the basis of ensuring the normal implementation of the function,the new real-time computing platform speeds up the task development cycle,reduces repeated development,and achieves a higher Resource utilization efficiency and higher task throughput.