The Design And Implementation Of General Scheduler’s Front End Based On Google Closure

With the rapid development of Internet products, Baidu has accumulated a lot of offline data. Building these data requires large numbers of ETL jobs. At the same time, many investigation jobs need to be established to make use of the data for producing a veriaty of reports. All of the ETL and investigation jobs need to be managed by a scheduling system. So how to effectively complete job management and scheduling and adapt to different requirements has become an important research topic.This paper firstly studies the industry-leading scheduling systems, and puts forward the important characteristics that a general scheduling system should provide based on the actual situation of Baidu. And then it introduces related technologies such as Thrift framework and Google Closure tools. Afterwards the paper analyzes the main requirements of the system, and makes the design of overall architecture and module division based on the requirements. The paper details the design and implementation of the front end of the system. Finally it summarizes all the work done, and looks forward to the future of the system.The general scheduler introduced in this paper has achieved well scalability and generality in addition to high stability and efficiency. The system would meet different kinds of requirements. It consists of a front-end server and a back-end server. The front-end server is responsible for page displaying and user interaction, which concerns about the system’s usability and user experience. While the back-end server takes charge of job management, scheduling and actual executions, and emphasizes the stability and efficiency. The communication between them are completed through Thrift service calls.The system is now running and providing a unified scheduling service for every product line, meeting a variety of scheduling requirements, and it gains nice praise.