The Design And Implementation Of Data Warehouse Task Scheduling Management Platform

The task scheduling management platform for data warehouse can be applied to the construction of big data projects.In the face of the increasing application of big data services,the application of data science and big data is increasing,the number of heterogeneous systems is gradually increasing,the response frequency is gradually promoting,and requirements for the management level is also improved accordingly.More and more pressure is on the data infrastructure platform,posing new challenges to these data infrastructure platforms.Therefore,the design and implementation of the task scheduling management platform for data warehouse is proposed.The task scheduling management platform for data warehouse was built above the operating system,Java virtual machine,and db2 database management system.It does not directly interface with the system hardware.This platform can access and operate the hardware through the basic system without direct hardware interface.This platform is a web application system.The coding elements in the platform mainly include JSP page files,Java program files,configuration files,Sql scripts,and so on.The platform is designed with two parts:the front-end WEB subsystem and the background scheduling engine subsystem.In the front-end WEB subsystem,basic scheduling,daily management,daily monitoring,and system management were developed under the JavaEE technology framework.The background scheduling engine subsystem adopted a three-layer structure,which consisted of four modules:Server,Agent,Job Encapsulation,and Common Components.The task scheduling management platform for data warehouse implemented four modules:basic scheduling,daily management,daily monitoring,and system management.Among them,basic scheduling module included five functions:system maintenance,operation group maintenance,operation maintenance,operation parameter maintenance,and batch maintenance.Daily management module included four functions:node resource configuration management,job concurrency control,date of manual switching service,and start-stop scheduling system.Daily monitoring modules included eleven functions:job group redoing monitoring,job group pausing and recovering monitoring,redoing manual job,job group monitoring,daily job monitoring,run queue monitoring,waiting queue monitoring,failure queue monitoring,abnormal operation query of job group,abnormal operation query of job,and running map.System management module included eight functions:organization management,department management,user management,association between organization and department,password management,role management,menu management,and authority management.The development of the platform can bring huge benefits to the data warehouse,and plays an increasingly important role in improving data quality,reducing operation and maintenance costs,and reducing consumption.The scheduling management platform affects the improvement of data quality and economic benefit in a direct or indirect way.