Design And Implementation Of Data Analysis Process Orchestration System

With the continuous promotion of big data concepts,more and more enterprises pay attention to data and accumulate a large amount of business data,the demand for decision-making based on data is growing rapidly.However,because of the lack of data analysis talents and the professional requirements of massive data analysis support technology,it is difficult for enterprises to quickly obtain effective data analysis capabilities,and it is urgent to get tools which can reduce the difficulty of data analysis implementation and support analysis and modeling personnel to easily and quickly organize data analysis processes,so that enable the empowerment of the enterprise.Therefore,from the perspective of ease of use and versatility,this thesis designs and implements a configurable data analysis process orchestration system,so that users do not have to pay attention to the underlying implementation details of data analysis operations,and can concentrate on the design of business data analysis.For this purpose,this paper adopts B/S architecture,uses SSM to form system development framework,and uses React to support front-end implementation.Based on Airflow orchestration engine and MySQL database system to realize task scheduling and data storage.We designed and implemented this system.The system is mainly composed of 6 core modules.The data import module is used for connection with the business database.The data preprocessing module provides multiple classic data processing methods.The program editing module supports the data analysis algorithm implemented by the Python script or the SQL statement.The model calling module is used to interface with the algorithm market platform of the project team where the author is located to realize the call of the algorithm model.The service publishing module uses the Docker container technology to realize the start and stop of the analysis process,and feedbacks the execution situation in real time.The service scheduling module sets the data monitoring and execution cycle for the analysis process.The system presets a variety of data processing and data analysis methods to achieve manual scheduling and operation monitoring of the data analysis process.The system has been applied to a national group company,which can meet the needs of the customized data analysis process in actual online use,and has been recognized and praised by the business staff.