Design And Implementation Of Integrated Data Management System Based On Spring Boot

With the development of enterprise business,the data of enterprise has expanded rapidly.The demand of the enterprise data has become increasingly complex to the manager and the operational staff.Faced with hundreds of database tables and T-unit-based data quantity in business department,how to calculate hundreds of indices of offline and online database under the time and space polymerization dimensions in the millisecond-level is the challenge of Integrated Data Management System.Except for the query and calculation of data,the manager expect to make a further data mining,aiming to support the smooth running of the business.This thesis designed a set of integrated management system for solving the above problems based on the offline and real-time data of micro services.The system employs the micro service system based on Spring Boot and Protobuf to implement the data interaction from the front to the back-end and manage the module system and data communication.Also,this system integrates the interface characteristics of the CAS system and data systems,which accomplishes the inter-service authentication and log management architecture of the data system and improves the security and availability.In the system,the offline data management service preprocesses offline data by using Crontab which do daily timed task to control tools such as Sqoop,Hive,Hbase,HDFS,etc.Meanwhile,it uses Kylin to pre-calculate and pre-aggregate the preprocessed data to deal with the demand of complex and high concurrency offline data.The online data management service applies timer-based Mysql task management to preprocess the data and perform ms-level data query responses.The system also uses machine learning time-series algorithm from machine learning to analyze the data.Using Holt-winters algorithm,combined with the actual characteristics of the data,the time series prediction based on the core data of the system is realized.After the development of this framework is completed,it has been applied to the udata systemthe of the SF City Business and fengsheng APP.The application of this framework solves the bottleneck problem that the data system cannot quickly respond to complex off-line and real-time data queries,and it provides key information for the company's operational status analysis and monitoring of change trends.it provides reliable assistance in the long-term growth and development of the company under the enterprise big data environment.